A method and system for fusing millimeter wave radar and visual recognition targets

Abstract

The application discloses a kind of millimeter wave radar and the fusion method and system of visual identification target, the method comprises: obtaining the data information of multiple first target points and multiple second target points;The first target point represents the target object detected by millimeter wave radar sensor, and the second target point represents the target object detected by camera;With the first target point as the center, the absolute threshold set is expanded longitudinally and transversely, to form the first confidence interval;With the second target point as the center, the relative threshold is expanded longitudinally and transversely to form the second confidence interval;The size of the relative threshold is linearly related to the distance and speed of the second target point;KM algorithm is executed to fuse, as many as possible millimeter wave radar detected first target points and camera detected second target points are one-to-one associated as the same target object.The method can improve the target fusion effect and reduce the generation of virtual target.

Description

Technical Field

[0001] This invention relates to the field of automatic emergency braking technology, and in particular to a method and system for fusing millimeter-wave radar with visual target recognition. Background Technology

[0002] Current Automatic Emergency Braking System (AEBS) designs mostly use radar and camera sensors to fuse the detected targets and determine the location of obstacles, such as... Figure 1 As shown, when a camera identifies a target, the target's speed and distance can significantly deviate from the camera's recognition. If camera and radar fusion fails to consider these deviations caused by motion or distance, it's possible that some objects identified by both radar and camera might not be fused together, resulting in false targets. Figure 2 As shown. Summary of the Invention

[0003] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a method and system for fusing millimeter-wave radar with visual target recognition, which can reduce the occurrence of false targets and improve the accuracy of target recognition.

[0004] To achieve the above objectives, the present invention provides a method for fusing millimeter-wave radar and visual target recognition, comprising the following steps:

[0005] Data information of multiple first target points and multiple second target points is acquired. The data information includes longitudinal distance information and lateral distance information of the target points, where longitudinal distance represents the vehicle's direction of travel. The first target points represent targets detected by millimeter-wave radar sensors, and the second target points represent targets detected by cameras.

[0006] A first confidence interval is formed by expanding longitudinally and laterally with the first target point as the center and using a set absolute threshold; a second confidence interval is formed by expanding longitudinally and laterally with the second target point as the center and using a relative threshold; the magnitude of the relative threshold is linearly related to the distance and speed of the second target point.

[0007] The KM algorithm is executed for fusion, associating as many first target points detected by millimeter-wave radar and second target points detected by camera as possible one-to-one as the same target object: based on the distance between the first target point and the second target point, from near to far, a first target point and a second target point are selected for comparison; if the first target point is in the second confidence interval of the second target point or the second target point is in the first confidence interval of the first target point, then the first target point and the second target point are considered to be the same target object.

[0008] Furthermore, the first confidence interval is based on the first target point (X). L ,Y L Centered on R δ Expand the rectangular area both vertically and horizontally.

[0009] Furthermore, the second confidence interval is based on the second target point (X). C, Y C Centered on (X), with (X) min ,Y min ), (X max ,Y max ) is a rectangular region with diagonal points, and (X) min ,Y min ), (X max ,Y max The value is obtained through the following formula:

[0010] X max =X C +X δ ×X C

[0011] X min =X C -X δ ×X C

[0012] Y max =Y C +Y δ ×Y C

[0013] Y min =Y C -Y δ ×Y C

[0014] Among them, X δ This represents the threshold for the percentage deviation of the camera's ranging on the X-axis; Y... δ This represents the threshold for the percentage deviation of the camera's ranging on the Y-axis.

[0015] Furthermore, the X δ Y δ As a constant, X δ The value is a certain value that is not less than the maximum value of the x-axis deviation percentage measurement within the set ranging interval; the Y... δ It is a value that is not less than the maximum value of the y-axis deviation percentage measurement within the set distance measurement interval.

[0016] Furthermore, the X δ Y δ The value of X is a variable. δ Yδ The calculation formula is:

[0017] X δ =K1×Y C +K3×V+C1

[0018] Y δ =K2×Y C +K4×V+C2

[0019] Among them, Y C V represents the longitudinal distance between the camera and the target, V represents the current vehicle speed, and K1, K2, K3, K4, C1, and C2 are fitting coefficients.

[0020] Furthermore, the fusion method also includes: combining the longitudinal distance, lateral distance, and velocity of the target object detected by the millimeter-wave radar and the category of the target object detected by the camera as target-level data information of the target object based on the fusion result.

[0021] The present invention also proposes a fusion system for millimeter-wave radar and visual target recognition, including millimeter-wave radar, camera and fusion processor; the fusion processor synchronously receives target information detected by the millimeter-wave radar and the camera, and executes the above-described fusion method for millimeter-wave radar and visual target recognition to associate the target detected by the millimeter-wave radar and the target detected by the camera.

[0022] This invention achieves the following technical effects: By studying the influence of speed and distance on the target recognition of cameras and radar, this invention sets the confidence interval of the target recognition by the camera according to the percentage of speed and distance. Thus, during KM fusion, the confidence interval of the target measured by the camera is adaptively adjusted according to distance and speed, thereby improving the target fusion effect and reducing the generation of false targets. Attached Figure Description

[0023] Figure 1 This is a diagram showing the positions of the vehicle's millimeter-wave radar, camera, and target.

[0024] Figure 2 This is a schematic diagram of the KM fusion results of existing vehicle millimeter-wave radar and camera target identification;

[0025] Figure 3 It is a test sample showing the relationship between the X-axis deviation value and the longitudinal distance of the radar target identification when the vehicle is stationary;

[0026] Figure 4 It is a test sample showing the relationship between the Y-axis deviation and longitudinal distance of the radar in identifying targets when the vehicle is stationary;

[0027] Figure 5It is a test sample of the relationship between the X-axis deviation value and the longitudinal distance of the radar in identifying the target when the vehicle is in motion;

[0028] Figure 6 It is a test sample showing the relationship between the Y-axis deviation value and the longitudinal distance of the radar in identifying the target when the vehicle is in motion;

[0029] Figure 7 It is a test sample showing the relationship between the X-axis deviation value and the longitudinal distance of the target identified by the camera;

[0030] Figure 8 It is a test sample showing the relationship between the Y-axis deviation value and the longitudinal distance of the target identified by the camera;

[0031] Figure 9 It is a test sample showing the relationship between the X-axis deviation value and the speed of the target identified by the camera;

[0032] Figure 10 It is a test sample showing the relationship between the Y-axis deviation value and the speed of the target identified by the camera;

[0033] Figure 11 This is a schematic diagram of the KM fusion of radar and camera for target identification according to the present invention;

[0034] Figure 12 This invention is a fusion system of millimeter-wave radar and visual target recognition. Detailed Implementation

[0035] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0036] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0037] This invention proposes a method for fusing millimeter-wave radar with visual target recognition, which is used to associate the target recognized by the vehicle's millimeter-wave radar with the target recognized by the vehicle camera.

[0038] First, the ranging performance of the vehicle-mounted millimeter-wave radar and camera was calibrated.

[0039] like Figures 3 to 6As shown, a typical vehicle-mounted millimeter-wave radar was selected, and its ranging error was calibrated within a set emergency braking range (e.g., 0-160 meters). Experimental calibration showed that, both when the vehicle was stationary and moving, the absolute values ​​of the X-axis deviation (vehicle yaw direction) and Y-axis deviation (vehicle travel direction) of the target detected by the radar were within 1 meter. These absolute deviation values ​​can be used to represent the X-axis and Y-axis deviation thresholds for radar ranging. Furthermore, within the set speed range, the influence of speed on radar ranging is negligible. Through statistical analysis of the experimental data, a confidence interval for the deviation value can be set, such as 95%, meaning that 95% of the radar test data has a deviation value less than the set deviation threshold. This deviation threshold can be set to R. δ =1 (unit: meter), assuming the target location identified by the radar is (X L ,Y L If the true target location is within the range of (X), then the confidence interval is defined as follows: L ,Y L Centered on R δ Expand the rectangular area both vertically and horizontally.

[0040] Similarly, as Figures 7 to 10 As shown, a vehicle-mounted camera was selected, and its ranging error was measured within a range of 0-160 meters. Experimental results showed that the camera's target recognition deviation increases with distance, and this deviation is best expressed as a percentage. In this embodiment, the percentage deviation of the target's X-axis and Y-axis is between 0-8%. Converted to absolute deviation, at a distance of 100 meters, the absolute deviation can reach several meters.

[0041] Therefore, during calibration, the maximum percentage of x-axis deviation of the camera or this type of camera within a set ranging range (e.g., 0-160 meters) can be calculated; a value not less than this maximum percentage of x-axis deviation can be selected as the x-axis deviation percentage threshold. Similarly, during calibration, the maximum percentage of y-axis deviation of the camera or this type of camera within a set ranging range (e.g., 0-160 meters) can be calculated; a value not less than this maximum percentage of y-axis deviation can be selected as the y-axis deviation percentage threshold.

[0042] like Figure 9 and Figure 10 As shown, vehicle speed also has some impact on the x-axis error and y-axis deviation of distance measurement, but the impact is relatively small.

[0043] Furthermore, such as Figures 7 to 10The experimental data shows that, affected by the algorithm's accuracy, the percentage deviation of camera ranging increases with the increase of distance and speed. At this time, by linear fitting, we can obtain the linear equations of the X-axis deviation percentage and Y-axis deviation percentage of camera ranging with distance, and the linear equations of the X-axis deviation percentage and Y-axis deviation percentage of camera ranging with speed. Through these two linear equations, we can determine the linear equations of the X-axis deviation percentage and Y-axis deviation percentage of camera ranging with distance and speed. Then, by setting the confidence level (such as 95% or 100%), we can adjust the constants in the linear equations to determine the threshold values ​​of the X-axis deviation percentage and Y-axis deviation percentage of camera ranging. This way, the thresholds are set with higher accuracy.

[0044] The linear equations between the X-axis deviation percentage threshold and the Y-axis deviation percentage threshold for camera ranging and the distance are as follows:

[0045] X1 = K1 × D + B1

[0046] Y1=K2×D+B2

[0047] Where D is the target distance, and K1, K2, B1, and B2 are all fitting constants, which are positive real numbers.

[0048] The linear equations for the X-axis and Y-axis deviation percentage thresholds and the velocity in camera ranging are as follows:

[0049] X2 = K3 × V + B3

[0050] Y2=K4×V+B4

[0051] Where V is the vehicle speed, and K3, K4, B3, and B4 are fitting constants, which are positive real numbers.

[0052] The linear equations between the X-axis deviation percentage threshold and the Y-axis deviation percentage threshold of the camera ranging and the distance and speed can be expressed as:

[0053] X δ =X1+X2=K1×D+K3×V+B5

[0054] Y δ =Y1+Y2=K2×D+K4×V+B6

[0055] Among them, B5 and B6 are fitting constants, which are positive real numbers.

[0056] from Figures 7 to 10It can be seen that in the longitudinal distance range of 0-160 meters, the percentage errors of the camera ranging on both the x-axis and y-axis gradually increase from about 2% to about 8%. In the range of vehicle speed of 10-100 km / h, the x-axis deviation of the camera ranging increases from 0.05 meters to about 0.22 meters; the y-axis deviation increases from 0.2 meters to 1.9 meters. Therefore, by fitting the curves and calculating, the parameters K1, K3, K2, and K4 are 3.2% / meter, 3.9% / meter, 0.018 hours / km, and 0.20 hours / km, respectively.

[0057] Assume the camera measures the target's position as (X). C, Y C If the true target location is within the range of (X), then the confidence interval is defined as follows: min ,Y min ), (X max ,Y max Let ) be a rectangular region with diagonal points, where:

[0058] X max =X C +X δ ×X C

[0059] X min =X C -X δ ×X C

[0060] Y max =Y C +Y δ ×Y C

[0061] Y min =Y C -Y δ ×Y C

[0062] Therefore, the fusion of camera and millimeter-wave radar needs to take into account the recognition deviation caused by factors such as motion or distance between the camera and radar. The goal is to identify the same target object by both the millimeter-wave radar and the camera, and then fuse them to form the same target object. The longitudinal distance, lateral distance and speed of the target object detected by the millimeter-wave radar, and the category of the target object detected by the camera are combined as the target-level data information of the target object.

[0063] Therefore, this invention proposes a method for fusing millimeter-wave radar and visual target recognition, comprising the following steps:

[0064] Data information of multiple first target points and multiple second target points is acquired. The data information includes longitudinal (y-axis) distance information and lateral (x-axis) distance information of the target points, where the longitudinal (y-axis) represents the vehicle's direction of travel; the first target points represent targets detected by millimeter-wave radar sensors, and the second target points represent targets detected by cameras;

[0065] A first confidence interval is formed by expanding longitudinally and laterally with the first target point as the center and using a set absolute threshold; a second confidence interval is formed by expanding longitudinally and laterally with the second target point as the center and using a relative threshold; the magnitude of the relative threshold is linearly related to the distance and speed of the second target point.

[0066] The KM algorithm is executed for fusion, associating as many first target points detected by millimeter-wave radar and second target points detected by cameras one-to-one as the same target object: based on the distance between the first target point and the second target point, from near to far, a first target point and a second target point are selected for comparison; if the first target point is within the second confidence interval of the second target point or the second target point is within the first confidence interval of the first target point, then the first target point and the second target point are considered to be the same target object;

[0067] Based on the fusion results, the longitudinal distance, lateral distance, and velocity of the target detected by the millimeter-wave radar, and the category of the target detected by the camera are combined to form the target-level data information of the target.

[0068] In this embodiment, the first confidence interval is based on the first target point (X). L, Y L Centered on R δ Expand the rectangular area both vertically and horizontally.

[0069] In this embodiment, the second confidence interval is based on the second target point (X). C, Y C Centered on (X), with (X) min ,Y min ), (X max ,Y max ) is a rectangular region with diagonal points, and (X) min ,Y min ), (X max ,Y max The value is obtained through the following formula:

[0070] X max =X C +X δ ×X C

[0071] Xmin =X C -X δ ×X C

[0072] Y max =Y C +Y δ ×Y C

[0073] Y min =Y C -Y δ ×Y C

[0074] Among them, X δ This represents the threshold for the percentage deviation of the camera's ranging on the X-axis; Y... δ This represents the threshold for the percentage deviation of the camera's ranging on the Y-axis.

[0075] Due to X δ Y δ Due to the combined effects of distance and vehicle speed, the accuracy of camera ranging decreases as both distance and speed increase. Therefore, X δ Y δ This can be further expressed as:

[0076] X δ =K1×Y C +K3×V+C1

[0077] Y δ =K2×Y C +K4×V+C2

[0078] Among them, Y C X represents the longitudinal distance between the camera and the target, V represents the current vehicle speed, and K1, K2, K3, K4, C1, and C2 are fitting coefficients. In the test data given in this embodiment, K1, K2, K3, K4, C1, and C2 are all positive real numbers. δ Y δ For values ​​without units, the units for K1 and K3 are " / meter", the units for K2 and K4 are "hours / kilometer", and the units for C1 and C2 are ununited.

[0079] like Figure 11 As shown, the first confidence interval for radar target identification is included within the second confidence interval for camera target identification. This allows it to be determined that the radar-identified target and the camera-identified target within the second confidence interval are the same target, thus enabling the fusion of their information. Figure 2Traditional fusion methods cannot merge targets identified at the same location into a single target. However, when using high-precision cameras or obtaining high-precision camera ranging values ​​through software optimization, at close range, the second confidence interval of the camera-identified target may be contained within the first confidence interval of the radar-identified target. This allows the radar-identified target and the camera-identified target within the first confidence interval to be determined as the same target, thus enabling the fusion of their information.

[0080] This invention studies the impact of speed and distance on target recognition by cameras and radar. It sets the confidence interval of target recognition by the camera according to the percentage of speed and distance. Thus, during KM fusion, the confidence interval of the target measured by the camera is adaptively adjusted according to distance and speed, thereby improving the target fusion effect and reducing the generation of false targets.

[0081] like Figure 12 As shown, this invention also proposes a fusion system for millimeter-wave radar and visual target recognition, including a millimeter-wave radar, a camera, and a fusion processor. The fusion processor synchronously receives target information detected by the millimeter-wave radar and the camera, and executes the aforementioned fusion method for millimeter-wave radar and visual target recognition, associating the target detected by the millimeter-wave radar and the target detected by the camera.

[0082] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for fusing millimeter-wave radar and visual target recognition, characterized in that: Includes the following steps: Data information of multiple first target points and multiple second target points is acquired. The data information includes longitudinal distance information and lateral distance information of the target points, where longitudinal distance represents the vehicle's direction of travel. The first target points represent targets detected by millimeter-wave radar sensors, and the second target points represent targets detected by cameras. Using the first target point as the center, expand vertically and horizontally with a set absolute threshold to form a first confidence interval; A second confidence interval is formed by longitudinally and laterally expanding a relative threshold with the second target point as the center; the magnitude of the relative threshold is linearly related to the distance and velocity of the second target point. The KM algorithm is executed for fusion, and the first target point detected by the millimeter-wave radar and the second target point detected by the camera are associated one-to-one as the same target object: according to the distance between the first target point and the second target point, from near to far, a first target point and a second target point are selected for comparison; if the first target point is within the second confidence interval of the second target point or the second target point is within the first confidence interval of the first target point, then the first target point and the second target point are considered to be the same target object; The first confidence interval is based on the first target point (X). L ,Y L Centered on R δ Expand the rectangular area both vertically and horizontally; The second confidence interval is based on the second target point (X). C ,Y C Centered on (X), with (X) min ,Y min ), (X max ,Y max ) is a rectangular region with diagonal points, and (X) min ,Y min ), (X max ,Y max The value is obtained through the following formula: X max =X C +X δ ×X C ; X min =X C -X δ ×X C ; AND max And C +And δ ×Y C ; AND min And C -AND δ ×Y C ; Among them, X δ This represents the threshold for the percentage deviation of the camera's ranging on the X-axis; Y... δ The threshold representing the percentage deviation of the camera's ranging distance along the Y-axis; The X δ Y δ The value of X is a variable. δ Y δ The calculation formula is: X δ =K1×Y C +K3×V+C1; AND δ =K2×Y C +K4×V+C2; Among them, Y C V represents the longitudinal distance between the camera and the target, V represents the current vehicle speed, and K1, K2, K3, K4, C1, and C2 are fitting coefficients.

2. The method for fusing millimeter-wave radar and visual target recognition as described in claim 1, characterized in that: The R δ The value ranges from 0.3 to 1 meter.

3. The method for fusing millimeter-wave radar and visual target recognition as described in claim 1, characterized in that: The fusion method further includes: based on the fusion result, combining the longitudinal distance, lateral distance, and velocity of the target object detected by the millimeter-wave radar with the category of the target object detected by the camera as target-level data information of the target object.

4. A fusion system for millimeter-wave radar and visual target recognition, characterized in that: The system includes a millimeter-wave radar, a camera, and a fusion system. The fusion system synchronously receives target information detected by the millimeter-wave radar and the camera, and executes the fusion method of millimeter-wave radar and visual target recognition as described in any one of claims 1-3 to associate the target detected by the millimeter-wave radar with the target detected by the camera.

Images