Data fusion method and device based on camera and radar, vehicle and medium

Abstract

The invention discloses a data fusion method and device based on a camera and a radar, a vehicle and a medium, and relates to the technical field of intelligent driving. The method comprises the following steps: performing image acquisition on a first lateral area by using a camera to obtain a lateral image; performing first target detection based on the lateral image to obtain initial lateral detection frame information, and performing position conversion on the initial lateral detection frame information to obtain target lateral detection frame information; performing second target detection on the second lateral area by using the millimeter wave radar to obtain initial lateral point cloud information, and performing position conversion on the initial lateral point cloud information to obtain target lateral point cloud information; performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result; and performing information fusion on the target lateral point cloud information and the target lateral detection frame information according to the obstacle source result as a first result. The accuracy of data fusion can be improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a method and device, a vehicle, and a medium for data fusion based on a camera and a radar. Background Art

[0002] In the automotive field, when a vehicle is driving on the road and a vehicle passes it from the side or it overtakes other vehicles, the vehicle's detection of the position, length, width, speed and other information of the other vehicle (hereinafter referred to as the obstacle vehicle) will fluctuate greatly, and even the obstacle vehicle may be lost. The reason is that the current single sensor (such as camera, millimeter wave radar) has poor detection of lateral obstacle vehicles.

[0003] The related art proposes to use a neural network model to fuse the visual information collected by the camera and the point cloud information collected by the millimeter-wave radar to obtain information such as position, length, width, and speed. However, on the side of the vehicle, the obstacle vehicle may pass through the detection areas of four sensors (such as the lower right camera, the lower right millimeter-wave radar, the upper right camera, and the upper right millimeter-wave radar). At this time, the collected visual information and point cloud information involve four detection areas, but they are all the same obstacle vehicle. It is difficult for the neural network model to effectively capture the differences and associations between visual information and point cloud information, resulting in poor accuracy of data fusion. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a data fusion method and device based on camera and radar, a vehicle, and a medium, which can improve the accuracy of data fusion.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a data fusion method based on a camera and a radar, the method comprising:

[0006] Using a camera pre-deployed on the target vehicle to collect an image of a first side area of ​​the target vehicle to obtain a side image;

[0007] Performing a first target detection based on the lateral image to obtain initial lateral detection frame information of the first obstacle vehicle, and performing position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information; wherein the first conversion matrix represents a conversion matrix from the camera to a preset reference position;

[0008] Using a millimeter-wave radar pre-deployed on the target vehicle to perform a second target detection on a second lateral area of ​​the target vehicle, initial lateral point cloud information of the second obstacle vehicle is obtained, and the initial lateral point cloud information is position-converted according to a preset second conversion matrix to obtain target lateral point cloud information; wherein the second conversion matrix represents a conversion matrix of the millimeter-wave radar to the reference position, and the first lateral area overlaps with the second lateral area;

[0009] Obstacle source detection is performed according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result; wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle;

[0010] The target lateral point cloud information and the target lateral detection box information are fused according to the first result.

[0011] In one embodiment, performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result includes:

[0012] Performing point cloud projection on the lateral image according to the point cloud in the target lateral point cloud information to obtain a target projection point cloud rectangular frame;

[0013] A correlation is calculated based on the target projection point cloud rectangular frame and the target lateral detection frame information to obtain a radar visual correlation;

[0014] The source results are divided according to the radar visual correlation to obtain the obstacle source result.

[0015] In one embodiment, performing point cloud projection on the lateral image according to the point cloud in the target lateral point cloud information to obtain a target projection point cloud rectangular frame includes:

[0016] Projecting the point cloud in the target lateral point cloud information onto the lateral image to obtain target projection pixel points;

[0017] A maximum enclosing rectangle is generated according to the target projection pixel points to obtain the target projection point cloud rectangular frame.

[0018] In one embodiment, the camera includes a front-end camera and a rear-end camera on the same side, and the lateral image includes a front lateral image captured by the front-end camera and a rear lateral image captured by the rear-end camera;

[0019] Before performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result, the method further includes:

[0020] Obstacle source detection is performed according to the target lateral detection frame information of the front lateral image and the target lateral detection frame information of the rear lateral image to obtain front and rear visual obstacle source marks; wherein the front and rear visual obstacle source marks include a first state, and the first state indicates that the obstacle vehicle detected from the front lateral image and the obstacle vehicle detected from the rear lateral image are the same vehicle;

[0021] According to the first state, the one with the smaller frame area between the front lateral image and the rear lateral image is deleted; wherein the frame area represents the area of ​​the target lateral detection frame corresponding to the target lateral detection frame information.

[0022] In one embodiment, the performing obstacle source detection according to the target lateral detection frame information of the front lateral image and the target lateral detection frame information of the rear lateral image to obtain front and rear visual obstacle source marks includes:

[0023] Performing detection frame projection on the rear lateral image according to the target lateral detection frame information of the front lateral image to obtain a target projected visual rectangular frame;

[0024] Calculating the correlation between the target projection visual rectangular frame and the target lateral detection frame information of the rear lateral image to obtain the front-rear visual correlation;

[0025] Obstacle sources are divided according to the front and rear visual correlation to obtain the front and rear visual obstacle source labels.

[0026] In one embodiment, the target lateral detection frame information includes reference vertex coordinates, a rectangular frame width, and a rectangular frame height of the target lateral detection frame;

[0027] The step of performing detection frame projection on the rear lateral image according to the target lateral detection frame information of the front lateral image to obtain a target projected visual rectangular frame includes:

[0028] Determine the vertices according to the reference vertex coordinates of the front lateral image, the rectangular frame width and the rectangular frame height, and determine the frame pixel vertices;

[0029] Projecting the frame pixel vertices onto the rear lateral image to obtain target projected pixel vertices;

[0030] The target projection pixel vertices are connected to obtain the target projection visual rectangular frame.

[0031] In one embodiment, after fusing the target lateral point cloud information and the target lateral detection frame information according to the first result, the method further includes:

[0032] Obtaining target fusion information to obtain the target obstacle vehicle's flag, obstacle vehicle position, obstacle vehicle length, and obstacle vehicle width; wherein the target fusion information refers to information obtained after the information fusion is performed, and the flag is used to indicate whether the lateral image contains a complete obstacle vehicle or does not contain a complete obstacle vehicle;

[0033] If the flag indicates that the lateral image contains a complete obstacle vehicle, the position of the obstacle vehicle is updated according to the length and width of the obstacle vehicle to obtain a tracking position of the obstacle vehicle;

[0034] The target obstacle vehicle is tracked according to the obstacle vehicle tracking position.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a data fusion device based on a camera and a radar, the device comprising:

[0036] A visual acquisition module, used to acquire an image of a first side area of ​​the target vehicle using a camera pre-deployed on the target vehicle to obtain a side image;

[0037] A visual detection module, configured to perform a first target detection based on the lateral image, obtain initial lateral detection frame information of the first obstacle vehicle, and perform position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information; wherein the first conversion matrix represents a conversion matrix from the camera to a preset reference position;

[0038] A radar detection module, configured to perform a second target detection on a second lateral area of ​​the target vehicle using a millimeter-wave radar pre-deployed on the target vehicle, obtain initial lateral point cloud information of a second obstacle vehicle, and perform position conversion on the initial lateral point cloud information according to a preset second conversion matrix to obtain target lateral point cloud information; wherein the second conversion matrix represents a conversion matrix of the millimeter-wave radar to the reference position, and the first lateral area overlaps with the second lateral area;

[0039] an obstacle source detection module, configured to perform obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information, and obtain an obstacle source result; wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle;

[0040] A radar vision fusion module is used to fuse the target lateral point cloud information and the target lateral detection box information according to the first result.

[0041] To achieve the above objectives, the third aspect of an embodiment of the present application proposes a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0043] The data fusion method and device based on camera and radar proposed in the present application, the vehicle, and the storage medium acquire a lateral image through the camera deployed on the target vehicle, and the initial lateral detection frame information of the first obstacle vehicle can be obtained from the lateral image, and the target lateral detection frame information can be obtained after the position conversion. The initial lateral point cloud information can also be detected by the millimeter wave radar deployed on the target vehicle, and the target lateral point cloud information can be obtained after the position conversion. The target lateral detection frame information and the target lateral point cloud information are different information relative to the same reference position. After the obstacle source detection is performed, it can be known whether the first obstacle vehicle and the second obstacle vehicle are the same vehicle, that is, if the obstacle detection result is the first result, it means that the target lateral detection frame information and the target lateral point cloud information are both derived from the same vehicle. Finally, according to the first result, the target lateral point cloud information and the target lateral detection frame information are fused to obtain accurate obstacle information.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a camera and radar based data fusion method provided in an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a vehicle overtaking sideways provided in an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of the installation position and field of view angle of the camera and millimeter wave radar provided in the embodiment of the present application;

[0048] Figure 4 is a schematic diagram of the detection positions of the camera and the millimeter wave radar provided in the embodiment of the present application;

[0049] Figure 5 is a schematic diagram of a front lateral image provided by an embodiment of the present application;

[0050] Figure 6 is another schematic diagram of the front-lateral image provided by an embodiment of the present application;

[0051] Figure 7 is a flow chart of a camera and radar based data fusion method provided by another embodiment of the present application;

[0052] Figure 8 is a schematic diagram of a front lateral image detected by a right front camera provided in an embodiment of the present application;

[0053] Fig. 9 is a schematic diagram of the rear part of an obstacle vehicle detected by a right rear-end camera provided in an embodiment of the present application;

[0054] Fig.10 is a schematic diagram of a target projection pixel point and a target projection point cloud rectangular frame provided in an embodiment of the present application;

[0055] Fig.11 is a schematic diagram of obstacle vehicle tracking based on the vehicle center point provided in an embodiment of the present application;

[0056] Fig.12 It is a structural diagram of a camera and radar based data fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] The camera-and-radar-based data fusion method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the camera-and-radar-based data fusion method, etc., but is not limited to the above forms.

[0061] The camera-and-radar based data fusion method, camera-and-radar based data fusion device, vehicle and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the camera-and-radar based data fusion method in the embodiments of the present application is described.

[0062] Please refer to Figure 1 , Figure 1 An optional flow chart of a method for data fusion based on a camera and a radar is disclosed. Figure 1 The method may include but is not limited to steps 101 to 105.

[0063] Step 101, using a camera pre-deployed on the target vehicle to collect images of a first side area of ​​the target vehicle to obtain a side image;

[0064] Step 102: Perform first target detection based on the lateral image to obtain initial lateral detection frame information of the first obstacle vehicle, and perform position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information;

[0065] Step 103, using the millimeter wave radar pre-deployed on the target vehicle to perform a second target detection on the second side area of ​​the target vehicle, obtaining initial lateral point cloud information of the second obstacle vehicle, and performing position conversion on the initial lateral point cloud information according to a preset second conversion matrix to obtain target lateral point cloud information;

[0066] Step 104, performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result, wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle;

[0067] Step 105: fuse the target lateral point cloud information and the target lateral detection frame information according to the first result.

[0068] In steps 101 to 105 shown in the embodiment of the present application, the camera deployed on the target vehicle collects and obtains a lateral image, from which the initial lateral detection frame information of the first obstacle vehicle can be obtained, and the target lateral detection frame information can be obtained after the position conversion. The millimeter-wave radar deployed on the target vehicle can detect and obtain the initial lateral point cloud information, and obtain the target lateral point cloud information after the position conversion. The target lateral detection frame information and the target lateral point cloud information are different information relative to the same reference position. After performing obstacle source detection, it can be known whether the first obstacle vehicle and the second obstacle vehicle are the same vehicle. Finally, if the obstacle detection result is the first result, it means that the target lateral detection frame information and the target lateral point cloud information are both derived from the same vehicle, and specifically may be from different / same parts of the same vehicle. Finally, according to the first result, the target lateral point cloud information and the target lateral detection frame information are fused to obtain accurate obstacle information.

[0069] Before describing steps 101 to 105, the target vehicle, the camera, and the millimeter-wave radar are first described.

[0070] Target vehicle: refers to a vehicle traveling on the road and equipped with a camera and millimeter-wave radar. Camera: usually refers to a device that captures images or videos, and is widely used in target detection, photography, video recording and other fields. Millimeter-wave radar: is a radar system that uses millimeter-wave bands for detection and measurement, and can be applied to the field of autonomous driving. Millimeter-wave bands usually refer to frequencies between 30GHz and 300GHz.

[0071] Reference Figure 2 , the target vehicle is driving in the middle lane of the road. The obstacle vehicle can overtake the target vehicle from the left lane of the road, which can be called left overtaking. The obstacle vehicle may also overtake the target vehicle from the right lane of the road, which can be called right overtaking.

[0072] A right front camera, a right rear camera, a right front millimeter-wave radar, and a right rear millimeter-wave radar can be deployed on the right side of the target vehicle. Similarly, a left front camera, a left rear camera, a left front millimeter-wave radar, and a left rear millimeter-wave radar can be deployed on the left side of the target vehicle.

[0073] The same side includes the same left side or the same right side. The above-mentioned right front camera and right rear camera can be respectively referred to as the front camera and rear camera on the same right side. The above-mentioned left front camera and left rear camera can be respectively referred to as the front camera and rear camera on the same left side. Similarly, the above-mentioned right front millimeter-wave radar and right rear millimeter-wave radar can be respectively referred to as the front millimeter-wave radar and rear millimeter-wave radar on the same right side. The above-mentioned left front millimeter-wave radar and left rear millimeter-wave radar can be respectively referred to as the front millimeter-wave radar and rear millimeter-wave radar on the same left side.

[0074] Reference Figure 3 , take the right side as an example to explain. Figure 3 The three obstacle vehicles shown in the figure are the same vehicle at different times. The installation position of the right front camera is the right front end of the target vehicle, which is recorded as position A. The right front camera has a first field of view (FOV), that is, the first right front camera line L A1 and the second right front camera line L A2 The installation position of the right rear camera is the right rear end of the target vehicle, which is recorded as position B. The right rear camera has a second field of view angle, that is, the first right rear camera line L B1 and the second right rear camera line L B2 The installation position of the right front millimeter wave radar is the right front end of the target vehicle, which is recorded as position C. The right front millimeter wave radar has a third field of view angle, that is, the first right front radar line L C1 and the second right front radar line L C2 The installation position of the right rear end millimeter wave radar is the right rear end of the target vehicle, which is recorded as position D. The right rear end millimeter wave radar has a fourth field of view angle, that is, the first right rear end radar line L D1 and the second right rear radar line L D2 The angle formed.

[0075] It should be noted that the camera and millimeter-wave radar deployed on the left are similar to those deployed on the right, and will not be repeated here. In addition, although the camera and millimeter-wave radar are installed at the same end, such as the right front end or the right rear end, the actual installation positions are not necessarily the same / coincident, and some differences are allowed.

[0076] Continue to refer to Figure 3 Taking right overtaking as an example, the obstacle vehicle will continuously pass through the right side area of ​​the target vehicle. After discretizing the obstacle vehicles at multiple consecutive moments, we can get the obstacle vehicle at the bottom at time T1, the obstacle vehicle in the middle at time T2, and the obstacle vehicle at the top at time T3. At each moment, the right front camera and the right rear camera collect images, and the right front millimeter wave radar and the right rear millimeter wave radar collect point clouds.

[0077] The detection position accuracy of a single sensor is very poor, because they may not be in the sensor's advantageous area. False detection may also occur. For example, the obstacle vehicle at time T2 will pass through the detection areas of four sensors, namely the right rear camera, the right rear millimeter-wave radar, the right front camera and the right front millimeter-wave radar. Inaccurate detection positions of different sensors may result in a situation where there is actually only one obstacle vehicle, but multiple obstacle vehicles are detected, resulting in false detection. Moreover, the area directly to the side of the target vehicle belongs to the blind area of ​​the corner radar, which often causes missed detections. Among them, the area directly to the side refers to the area on the side where the obstacle vehicle and the target vehicle are in parallel, which can include the area directly to the right and the area directly to the left. In addition, referring to Figure 4 Based on the image captured by the camera, the position detection can be performed to obtain the first obstacle vehicle in the upper right corner represented by the dotted line, but the distance between the first obstacle vehicle and the target vehicle is unknown, that is, there is no depth direction. Based on the millimeter wave radar detection, the distance between the second obstacle vehicle and the target vehicle can be obtained. For example, based on Figure 4 The point cloud information corresponding to the 6 point clouds shown can be analyzed to obtain the distance between the second obstacle vehicle and the target vehicle. However, it is impossible to know whether the first obstacle vehicle and the second obstacle vehicle are the same vehicle. It can be seen that there is a large gap between the camera detection position and the millimeter-wave radar detection position, so it is difficult to associate them together. Considering all the problems, various detection problems in the side area of ​​the target vehicle will cause the target vehicle to be unable to make more accurate decisions and trajectory planning based on the movement of the obstacle vehicle appearing on the side. Steps 101 to 105 proposed in this application can well solve the various problems raised above.

[0078] In step 101 of some embodiments, a camera pre-deployed on the target vehicle is used to capture an image of the first lateral area of ​​the target vehicle to obtain a lateral image. For example, assuming that the first lateral area is the right lateral area, the right front end camera can be used to capture images to obtain a front lateral image, and the right rear end camera can be used to capture images to obtain a rear lateral image. At this time, the front lateral image can be called a right front lateral image, and the rear lateral image can be called a right rear lateral image. For another example, assuming that the first lateral area is the left lateral area, the left front end camera can be used to capture images to obtain a front lateral image, and the left rear end camera can be used to capture images to obtain a rear lateral image. At this time, the front lateral image can be called a left front lateral image, and the rear lateral image can be called a left rear lateral image.

[0079] In step 102 of some embodiments, a first target detection is performed based on the lateral image to obtain the initial lateral detection frame information of the first obstacle vehicle, and the initial lateral detection frame information is converted according to the preset first conversion matrix to obtain the target lateral detection frame information. The lateral image can be input into the visual detection model to obtain the initial lateral detection frame information of the first obstacle vehicle. The visual detection model can use the yo l ov8 model. The initial lateral detection frame information is visual information based on the camera. In combination with the above, the visual information obtained by the camera detection needs to be fused with the point cloud information obtained by the millimeter wave radar detection, and the installation position of the camera is not consistent with the installation position of the millimeter wave radar. Therefore, it is impossible to directly perform information fusion based on the initial lateral detection frame information. This embodiment introduces the first conversion matrix to convert the initial lateral detection frame information. Since the first conversion matrix represents the conversion matrix from the camera to the preset reference position, after the position conversion, the target lateral detection frame information obtained is visual information based on the reference position. Moreover, the subsequent steps will perform position conversion on the information detected by the millimeter-wave radar based on the second conversion matrix. Since the second conversion matrix represents the conversion matrix of the millimeter-wave radar to the reference position, point cloud information based on the reference position can be obtained after the position conversion, so as to perform information fusion based on the converted visual information and point cloud information.

[0080] It should be noted that the reference position can be the position of the center of the rear axle of the target vehicle. Generally, after the target vehicle leaves the factory, the installation angle and installation position of each sensor relative to the center of the rear axle of the vehicle are fixed, so the conversion matrix is ​​fixed. One sensor corresponds to one conversion matrix. That is, the right front camera, the right rear camera, the left front camera and the left rear camera correspond to different first conversion matrices respectively. The right front millimeter-wave radar, the right rear millimeter-wave radar, the left front millimeter-wave radar and the left rear millimeter-wave radar correspond to different second conversion matrices respectively.

[0081] In one example, the front side image can be obtained by collecting data using the left front camera. The front side image can be Figure 5As shown. The front lateral image can be input into the yo l ov8 model for the first target detection to obtain the initial lateral detection frame information {x, y, id, w, h, class, confidence} and flag of the first obstacle vehicle. x and y represent the horizontal and vertical coordinates of the initial lateral detection frame, respectively, id represents the initial identifier of the initial lateral detection frame, w represents the width of the initial lateral detection frame, h represents the height of the initial lateral detection frame, and class represents the type of the first obstacle vehicle contained in the initial lateral detection frame, such as a car, a truck, a tricycle, etc. conf i dence represents the confidence of the initial lateral detection frame. The flag is used to indicate whether the lateral image contains a complete obstacle vehicle or does not contain a complete obstacle vehicle, or in other words, the flag is used to indicate whether the obstacle vehicle is completely within the field of view of the lateral image. For example, in Figure 5 In , the initial lateral detection frame does not intersect / overlap with the boundary of the front lateral image. At this time, the flag indicates that the front lateral image contains the complete obstacle vehicle. For another example, in Figure 6 In the example, the boundary of the initial lateral detection frame intersects / overlaps with the boundary of the front lateral image. At this time, the flag indicates that the front lateral image does not contain the complete obstacle vehicle.

[0082] The first transformation matrix and the second transformation matrix can both be shown as follows:

[0083]

[0084] Among them, the 3×3 matrix represented by R11 to R13 is used to convert the installation angle, and Tx, Ty and Tz are used to convert the installation coordinates. It should be noted again that the first conversion matrix and the second conversion matrix can be obtained through calibration after the target vehicle is installed with a camera and a millimeter-wave radar, respectively. The specific values ​​of each element in the first conversion matrix and the second conversion matrix are determined according to the actual installation angle and the installation coordinates, and the embodiments of the present application do not specifically limit this.

[0085] In step 103 of some embodiments, a millimeter-wave radar pre-deployed on the target vehicle is used to perform second target detection on the second side area of ​​the target vehicle to obtain initial lateral point cloud information of the second obstacle vehicle, and the initial lateral point cloud information is position-converted according to a preset second conversion matrix to obtain target lateral point cloud information. The initial lateral point cloud information includes information such as the initial distance from the second obstacle vehicle to the millimeter-wave radar, the initial distance change rate, the initial azimuth, and the initial height. After the position conversion, the obtained target lateral point cloud information includes information such as the target distance from the second obstacle vehicle to the reference position, the target distance change rate, the target azimuth, and the target height.

[0086] In combination with the above, the second conversion matrix represents the conversion matrix of the millimeter wave radar to the reference position. The first lateral area overlaps with the second lateral area. For example, if the first lateral area is the left lateral area, then the second lateral area is also the left lateral area, but the two areas do not necessarily completely overlap, and there can be overlapping areas / intersecting areas. For example, if the first lateral area is the right lateral area, then the second lateral area is also the right lateral area. Similarly, the two areas do not necessarily completely overlap, and there can be overlapping areas / intersecting areas.

[0087] The process of performing position transformation on the initial lateral point cloud information according to the second transformation matrix is ​​similar to the process of performing position transformation on the initial lateral detection frame information according to the first transformation matrix, and will not be repeated here.

[0088] In one embodiment, before step 104, refer to Figure 7 , the data fusion method based on camera and radar can also include the following steps:

[0089] Step 701, performing obstacle source detection according to the target lateral detection frame information of the front lateral image and the target lateral detection frame information of the rear lateral image, and obtaining front and rear visual obstacle source marks; wherein the front and rear visual obstacle source marks include a first state;

[0090] Step 702: According to the first state, delete the one with the smaller frame area between the front lateral image and the rear lateral image.

[0091] Specifically, the front lateral image may be an image captured by the right front camera, and the rear lateral image is an image captured by the right rear camera. The front lateral image may also be an image captured by the left front camera, and the rear lateral image is an image captured by the left rear camera. The right front camera and the left front camera may be collectively referred to as the front camera. The right rear camera and the left rear camera may be collectively referred to as the rear camera.

[0092] In one example, refer to Figure 8 , the right front camera can capture the front part of the obstacle vehicle in the area to the right of the target vehicle and obtain the front side image. Fig. 9 The right rear camera can capture the rear part of the obstacle vehicle in the area directly to the right of the target vehicle and obtain a rear lateral image.

[0093] In addition to the first state, the front and rear visual obstacle source mark also includes a second state. The first state indicates that the obstacle vehicle detected from the front side image and the obstacle vehicle detected from the rear side image are the same vehicle. The second state indicates that the obstacle vehicle detected from the front side image and the obstacle vehicle detected from the rear side image are different vehicles.

[0094] In one embodiment, step 701 includes:

[0095] According to the target lateral detection frame information of the front lateral image, the detection frame is projected on the rear lateral image to obtain the target projection visual rectangular frame;

[0096] The correlation is calculated based on the target projection visual rectangular frame and the target lateral detection frame information of the rear lateral image to obtain the front and rear visual correlation;

[0097] Obstacle sources are divided according to the front and rear visual correlation to obtain front and rear visual obstacle source labels.

[0098] Specifically, the target lateral detection frame information includes the reference vertex coordinates, rectangular frame width and rectangular frame height of the target lateral detection frame. For example, the reference vertex coordinates are (x1, y1), the rectangular frame width is W, and the rectangular frame height is H. The reference vertex coordinates can select the coordinates of the upper left corner vertex of the target lateral detection frame.

[0099] In one embodiment, the step of projecting the detection frame on the rear side image according to the target side detection frame information of the front side image to obtain the target projected visual rectangular frame may specifically include:

[0100] Vertex determination is performed according to the reference vertex coordinates of the front lateral image, the rectangular frame width, and the rectangular frame height to determine the frame pixel vertex;

[0101] Project the frame pixel vertices to the rear lateral image to obtain the target projected pixel vertices;

[0102] The target projection pixel vertices are connected to obtain the target projection visual rectangular box.

[0103] In one example, the reference vertex coordinates detected from the right front lateral image are (x1, y1), the rectangular frame width is W, and the rectangular frame high position H is used to determine the vertices, and four frame pixel vertices are obtained, namely, the upper left frame pixel vertex P1 (x1, y1), the upper right frame pixel vertex P2 (x2, y2), the lower left frame pixel vertex P3 (x3, y3), and the lower right frame pixel vertex P4 (x4, y4). Among them, x2 = x1 + W, y2 = y1, x3 = x1, y3 = y1-H, x4 = x1 + W, y4 = y1-H. The four frame pixel vertices are projected into the right rear lateral image to obtain four target projection pixel vertices, namely, the upper left projection pixel vertex P1+, the upper right projection pixel vertex P2+, the lower left projection pixel vertex P3+, and the lower right projection pixel vertex P4+. After connection, the target projection visual rectangular frame can be obtained.

[0104] After obtaining the target projection visual rectangular frame, the step of performing correlation calculation based on the target projection visual rectangular frame and the target lateral detection frame information of the rear lateral image to obtain the front-to-back visual correlation may include: determining the target lateral rectangular frame contained in the rear lateral image based on the reference vertex coordinates, rectangular frame width and rectangular frame height of the rear lateral image; performing intersection-over-union calculation based on the target projection visual rectangular frame and the target lateral rectangular frame contained in the rear lateral image to obtain the front-to-back visual correlation. The intersection-over-union calculation refers to calculating the ratio between the intersection of two frames and the union of the two frames, that is, the front-to-back visual correlation may also be called IOU value (Intersection over Union).

[0105] After obtaining the front-to-back visual correlation, the above step of dividing the obstacle sources according to the front-to-back visual correlation to obtain the front-to-back visual obstacle source mark may include: comparing the front-to-back visual correlation with a preset visual correlation threshold; if the front-to-back visual correlation is less than or equal to the visual correlation threshold, determining that the front-to-back visual obstacle source mark is in the second state; if the front-to-back visual correlation is greater than the visual correlation threshold, determining that the front-to-back visual obstacle source mark is in the first state. For example, the second state may be represented as 0, and the first state may be represented as 1.

[0106] In step 702, if it is determined that the source flags of the front and rear visual obstacles are in the first state, it means that the obstacle vehicle in the front side image and the obstacle vehicle in the rear side image are the same vehicle, and the one with the smaller frame area in the front side image and the rear side image can be deleted. The frame area represents the area of ​​the target side detection frame corresponding to the target side detection frame information.

[0107] For example, the area of ​​the target lateral detection frame calculated according to the rectangular frame width and rectangular frame height included in the front lateral image is s1, and the area of ​​the target lateral detection frame calculated according to the rectangular frame width and rectangular frame height included in the rear lateral image is s2, and s1 is greater than s2, then the rear lateral image is deleted, so that only the front lateral image needs to perform step 104. The unit of the area can be square centimeters.

[0108] It is understandable that if the front and rear visual obstacle source marks are determined to be in the second state, it means that the obstacle vehicle in the front lateral image and the obstacle vehicle in the rear lateral image are different vehicles, and step 104 needs to be performed for the front lateral image and the rear lateral image respectively.

[0109] In step 104 of some embodiments, obstacle source detection is performed based on the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result. The obstacle source result includes a first result or a second result. The first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle. The second result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are different vehicles.

[0110] In one embodiment, step 104 may include:

[0111] Perform point cloud projection on the lateral image according to the point cloud in the target lateral point cloud information to obtain a target projection point cloud rectangular frame;

[0112] The correlation is calculated based on the target projection point cloud rectangular frame and the target lateral detection frame information to obtain the radar visual correlation;

[0113] The source results are divided according to the radar visual correlation to obtain the obstacle source results.

[0114] In one embodiment, the step of performing point cloud projection on the lateral image based on the point cloud in the target lateral point cloud information to obtain the target projection point cloud rectangular frame may include: projecting the point cloud in the target lateral point cloud information onto the lateral image to obtain the target projection pixel points; generating the maximum enclosing rectangle based on the target projection pixel points to obtain the target projection point cloud rectangular frame. Fig.10 After projecting the point cloud in the target lateral point cloud information onto the lateral image, 8 target projection pixel points can be obtained. Then, a maximum enclosing rectangle is generated according to the 8 target projection pixel points to obtain a target projection point cloud rectangular frame. This embodiment can simplify the generation process of the target projection point cloud rectangular frame, thereby improving data fusion efficiency.

[0115] It should be noted that although Fig.10 Eight target projection pixel points are shown, but the specific number is determined according to actual needs and is not specifically limited in this embodiment.

[0116] In one embodiment, the above-mentioned correlation calculation based on the target projection point cloud rectangular frame and the target lateral detection frame information to obtain the radar visual correlation includes: determining the target lateral rectangular frame included in the lateral image according to the reference vertex coordinates, rectangular frame width and rectangular frame height contained in the target lateral detection frame information; performing intersection and union ratio calculation based on the target projection point cloud rectangular frame and the target lateral rectangular frame included in the lateral image to obtain the radar visual correlation. The intersection and union ratio calculation has been explained above and will not be repeated here. This embodiment can simplify the calculation process of the radar visual correlation, thereby improving the efficiency of data fusion.

[0117] In one embodiment, the step of dividing the source results according to the radar visual correlation to obtain the obstacle source result may include: comparing the radar visual correlation with a preset radar visual correlation threshold; if the radar visual correlation is less than or equal to the radar visual correlation threshold, determining the obstacle source result to be the second result; if the radar visual correlation is greater than the radar visual correlation threshold, determining the obstacle source result to be the first result. For example, the second result may be represented as 0, and the first result may be represented as 1.

[0118] In step 105 of some embodiments, if the obstacle source result is the first result, it means that the first obstacle vehicle and the second obstacle vehicle are the same vehicle. At this time, the target lateral point cloud information and the target lateral detection frame information can be fused. For example, the target lateral point cloud information includes information such as the target distance from the second obstacle vehicle to the reference position, the target distance change rate, the target orientation, and the target height, and the target lateral detection frame information includes information such as the orientation from the first obstacle vehicle to the reference position, the vehicle type, the obstacle vehicle length, the obstacle vehicle width, and the flag position. After the information fusion is performed, accurate target fusion information can be obtained, including information such as the flag position, the obstacle vehicle speed, the obstacle vehicle position, the obstacle vehicle type, the obstacle vehicle length, and the obstacle vehicle width.

[0119] It can be understood that the target fused information refers to the information obtained after information fusion. The flag bit is used to indicate whether the lateral image contains a complete obstacle vehicle or does not contain a complete obstacle vehicle.

[0120] In one embodiment, after step 105, the camera-radar based data fusion method further includes:

[0121] Obtain the target fusion information to obtain the target obstacle vehicle's sign position, obstacle vehicle position, obstacle vehicle length and obstacle vehicle width;

[0122] If the flag indicates that the side image contains a complete obstacle vehicle, the position of the obstacle vehicle is updated according to the length and width of the obstacle vehicle to obtain the tracking position of the obstacle vehicle;

[0123] The target obstacle vehicle is tracked according to the obstacle vehicle tracking position.

[0124] Specifically, no matter the target vehicle overtakes the target obstacle vehicle, or the target obstacle vehicle overtakes the target vehicle, the following will occur: Fig.11There are three situations, that is, the first vehicle, the second vehicle, and the third vehicle are the same target obstacle vehicle at TI, T2, and T3 respectively. The first box K1, the second box K2, and the third box K3 represent the position of the first vehicle, the position of the second vehicle, and the position of the third vehicle observed by the target vehicle, respectively. The first position g1, the second position g2, and the third position g3 represent the position of the point closest to the target vehicle in the position of the first vehicle, the position of the point closest to the target vehicle in the position of the second vehicle, and the position of the point closest to the target vehicle in the position of the third vehicle, respectively. The position of the nearest point observed by the target vehicle can also be expressed as the nearest position that the millimeter-wave radar can detect the target obstacle vehicle. The length and width of the target obstacle vehicle are fixed values. For example, the coordinates of the first position g1 are (x T11 ,y T11 ), then the coordinates of the center point position g1+ of the first vehicle are (x T12 ,y T12 ), where x T12 =x T11 +length / 2,x T12 =y T11 –width / 2, length represents the length of the target obstacle vehicle, width represents the width of the target obstacle vehicle. For another example, the coordinates of the second position g2 are (x T21 ,y T21 ), then the coordinates of the center point position g2+ of the second vehicle are (x T22 ,y T22 ), where x T22 =x T21 , x T22 =y T21 –width / 2. For another example, the coordinates of the third position g3 are (x T31 ,y T31 ), then the coordinates of the center point position g3+ of the third vehicle are (x T32 ,y T32 ), where x T32 =x T31 –length / 2,x T32 =y T31 –wi dth / 2. It should be noted that Fig.11 Here, g2 and g2+ represent the same position.

[0125] When tracking a vehicle, this embodiment first returns the tracking point to the center point of the target obstacle vehicle to ensure that the tracking point is the coordinate position of the center point of the same target obstacle vehicle, so that the movement of the obstacle vehicle can be tracked more stably and accurately.

[0126] In one embodiment, the target obstacle vehicle moving from far to near or from near to far can be detected by the first target detection to output the flag position of whether the complete obstacle vehicle is completely detected, so as to determine the length and width of the obstacle vehicle, and then all the initial identifications of the same obstacle vehicle are updated to the tracking identification. When the target obstacle vehicle moves parallel to the target vehicle, the initial lateral point cloud information and the initial lateral detection frame information at the current moment can be used to associate with the tracking identification, and the target fusion information can be obtained by using the front and rear visual matching (i.e. step 701) and radar visual matching (i.e. step 104), and the obstacle vehicle is tracked and the current obstacle vehicle position is updated, which can prevent missed detection and overlapping obstacles.

[0127] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0128] The present application also discloses a data fusion device based on a camera and a radar, referring to Fig.12 The camera and radar based data fusion device includes: a visual acquisition module 1201, a visual detection module 1202, a radar detection module 1203, an obstacle source detection module 1204 and a radar visual fusion module 1205. The visual acquisition module 1201 is used to use a camera pre-deployed on the target vehicle to acquire an image of the first side area of ​​the target vehicle to obtain a side image;

[0129] The visual detection module 1202 is used to perform a first target detection based on the lateral image, obtain initial lateral detection frame information of the first obstacle vehicle, and perform position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information; wherein the first conversion matrix represents a conversion matrix from a camera to a preset reference position;

[0130] The radar detection module 1203 is used to perform a second target detection on a second side area of ​​the target vehicle using a millimeter wave radar pre-deployed on the target vehicle, obtain initial side point cloud information of the second obstacle vehicle, and perform position conversion on the initial side point cloud information according to a preset second conversion matrix to obtain target side point cloud information; wherein the second conversion matrix represents a conversion matrix of the millimeter wave radar to a reference position, and the first side area overlaps with the second side area;

[0131] The obstacle source detection module 1204 is used to perform obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result; wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle;

[0132] The radar vision fusion module 1205 is used to fuse the target lateral point cloud information and the target lateral detection box information according to the first result.

[0133] In one embodiment, before performing obstacle source detection based on the target lateral point cloud information and the target lateral detection frame information to obtain the obstacle source result, the camera and radar based data fusion device further includes a front and rear visual matching module for:

[0134] Obstacle source detection is performed according to the target side detection frame information of the front side image and the target side detection frame information of the rear side image to obtain front and rear visual obstacle source marks; wherein the front and rear visual obstacle source marks include a first state, and the first state indicates that the obstacle vehicle detected from the front side image and the obstacle vehicle detected from the rear side image are the same vehicle;

[0135] According to the first state, the one with the smaller frame area between the front lateral image and the rear lateral image is deleted; wherein the frame area represents the area of ​​the target lateral detection frame corresponding to the target lateral detection frame information.

[0136] In one embodiment, after fusing the target lateral point cloud information and the target lateral detection frame information according to the first result, the camera and radar based data fusion device further includes an obstacle tracking module for:

[0137] Obtain the target fusion information to obtain the target obstacle vehicle's flag position, obstacle vehicle position, obstacle vehicle length and obstacle vehicle width; wherein the target fusion information refers to the information obtained after information fusion, and the flag position is used to indicate whether the lateral image contains a complete obstacle vehicle or does not contain a complete obstacle vehicle;

[0138] If the flag indicates that the side image contains a complete obstacle vehicle, the position of the obstacle vehicle is updated according to the length and width of the obstacle vehicle to obtain the tracking position of the obstacle vehicle;

[0139] The target obstacle vehicle is tracked according to the obstacle vehicle tracking position.

[0140] It should be noted that the specific implementation of the camera-and-radar-based data fusion device is basically the same as the specific implementation of the camera-and-radar-based data fusion method described above, and will not be repeated here.

[0141] An embodiment of the present application also discloses a vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned camera and radar-based data fusion method is implemented.

[0142] It should be noted that the camera and lidar are installed on the vehicle.

[0143] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned camera and radar-based data fusion method.

[0144] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0150] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0155] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data fusion method based on camera and radar, characterized in that: The method comprises: Using a camera pre-deployed on the target vehicle to collect an image of a first side area of ​​the target vehicle to obtain a side image; Performing a first target detection based on the lateral image to obtain initial lateral detection frame information of the first obstacle vehicle, and performing position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information; wherein the first conversion matrix represents a conversion matrix from the camera to a preset reference position; Using a millimeter-wave radar pre-deployed on the target vehicle to perform a second target detection on a second lateral area of ​​the target vehicle, initial lateral point cloud information of the second obstacle vehicle is obtained, and the initial lateral point cloud information is position-converted according to a preset second conversion matrix to obtain target lateral point cloud information; wherein the second conversion matrix represents a conversion matrix of the millimeter-wave radar to the reference position, and the first lateral area overlaps with the second lateral area; Obstacle source detection is performed according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result; wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle; The target lateral point cloud information and the target lateral detection box information are fused according to the first result.

2. The method according to claim 1, characterized in that The performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result includes: Performing point cloud projection on the lateral image according to the point cloud in the target lateral point cloud information to obtain a target projection point cloud rectangular frame; A correlation is calculated based on the target projection point cloud rectangular frame and the target lateral detection frame information to obtain a radar visual correlation; The source results are divided according to the radar visual correlation to obtain the obstacle source result.

3. The method according to claim 2, characterized in that The step of performing point cloud projection on the lateral image according to the point cloud in the target lateral point cloud information to obtain a target projection point cloud rectangular frame includes: Projecting the point cloud in the target lateral point cloud information onto the lateral image to obtain target projection pixel points; A maximum enclosing rectangle is generated according to the target projection pixel points to obtain the target projection point cloud rectangular frame.

4. The method according to any one of claims 1 to 3, characterized in that: The camera includes a front-end camera and a rear-end camera on the same side, and the lateral image includes a front lateral image captured by the front-end camera and a rear lateral image captured by the rear-end camera; Before performing obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information to obtain an obstacle source result, the method further includes: Obstacle source detection is performed according to the target lateral detection frame information of the front lateral image and the target lateral detection frame information of the rear lateral image to obtain front and rear visual obstacle source marks; wherein the front and rear visual obstacle source marks include a first state, and the first state indicates that the obstacle vehicle detected from the front lateral image and the obstacle vehicle detected from the rear lateral image are the same vehicle; According to the first state, the one with the smaller frame area between the front lateral image and the rear lateral image is deleted; wherein the frame area represents the area of ​​the target lateral detection frame corresponding to the target lateral detection frame information.

5. The method according to claim 4, characterized in that The performing obstacle source detection according to the target lateral detection frame information of the front lateral image and the target lateral detection frame information of the rear lateral image to obtain front and rear visual obstacle source marks includes: Performing detection frame projection on the rear lateral image according to the target lateral detection frame information of the front lateral image to obtain a target projected visual rectangular frame; Calculating the correlation between the target projection visual rectangular frame and the target lateral detection frame information of the rear lateral image to obtain the front-rear visual correlation; Obstacle sources are divided according to the front and rear visual correlation to obtain the front and rear visual obstacle source labels.

6. The method according to claim 5, characterized in that The target lateral detection frame information includes reference vertex coordinates, rectangular frame width and rectangular frame height of the target lateral detection frame; The step of performing detection frame projection on the rear lateral image according to the target lateral detection frame information of the front lateral image to obtain a target projected visual rectangular frame includes: Determine the vertices according to the reference vertex coordinates of the front lateral image, the rectangular frame width and the rectangular frame height, and determine the frame pixel vertices; Projecting the frame pixel vertices onto the rear lateral image to obtain target projected pixel vertices; The target projection pixel vertices are connected to obtain the target projection visual rectangular frame.

7. The method according to any one of claims 1 to 3, characterized in that: After fusing the target lateral point cloud information and the target lateral detection frame information according to the first result, the method further includes: Obtaining target fusion information to obtain the target obstacle vehicle's flag, obstacle vehicle position, obstacle vehicle length, and obstacle vehicle width; wherein the target fusion information refers to information obtained after the information fusion is performed, and the flag is used to indicate whether the lateral image contains a complete obstacle vehicle or does not contain a complete obstacle vehicle; If the flag indicates that the lateral image contains a complete obstacle vehicle, the position of the obstacle vehicle is updated according to the length and width of the obstacle vehicle to obtain a tracking position of the obstacle vehicle; The target obstacle vehicle is tracked according to the obstacle vehicle tracking position.

8. A data fusion device based on camera and radar, characterized in that: The device comprises: A visual acquisition module, used to acquire an image of a first side area of ​​the target vehicle using a camera pre-deployed on the target vehicle to obtain a side image; A visual detection module, configured to perform a first target detection based on the lateral image, obtain initial lateral detection frame information of the first obstacle vehicle, and perform position conversion on the initial lateral detection frame information according to a preset first conversion matrix to obtain target lateral detection frame information; wherein the first conversion matrix represents a conversion matrix from the camera to a preset reference position; A radar detection module, configured to perform a second target detection on a second lateral area of ​​the target vehicle using a millimeter-wave radar pre-deployed on the target vehicle, obtain initial lateral point cloud information of a second obstacle vehicle, and perform position conversion on the initial lateral point cloud information according to a preset second conversion matrix to obtain target lateral point cloud information; wherein the second conversion matrix represents a conversion matrix of the millimeter-wave radar to the reference position, and the first lateral area overlaps with the second lateral area; an obstacle source detection module, configured to perform obstacle source detection according to the target lateral point cloud information and the target lateral detection frame information, and obtain an obstacle source result; wherein the obstacle source result includes a first result, and the first result is used to indicate that the first obstacle vehicle and the second obstacle vehicle are the same vehicle; A radar vision fusion module is used to fuse the target lateral point cloud information and the target lateral detection box information according to the first result.

9. A vehicle, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images