Rapid fusion method between radar data and visual target detection data

Through the improved D.IOU algorithm, the introduction of Euclidean distance and angle information into the fusion of radar data and visual target detection data, solves the problems of low matching accuracy and slow speed in the existing technology, and realizes more efficient data fusion and improves The target detection capability of autonomous vehicles.

CN119004371AActive Publication Date: 2024-11-22LIAONING UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411070812.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-22
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The prior art has low matching accuracy and slow speed in the fusion of radar data and visual target detection data, which cannot meet the requirements of timeliness and accuracy of high-speed vehicles.

Method used

The improved D.IOU algorithm is used to match the radar data and visual target detection data. By introducing Euclidean distance and angle information based on the original D.IOU, and determining the optimal weight and threshold through multiple experiments, a rough match is performed. and fine matching to improve matching accuracy and speed.

Benefits of technology

It significantly improves the accuracy and timeliness of data fusion, reduces matching time, improves the overall D.IOU average value of matching results, and reaches 0.2ms/frame. The performance of high-quality results is slightly reduced, but the overall effect is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004371A_ABST
    Figure CN119004371A_ABST
Patent Text Reader

Abstract

The invention discloses a rapid fusion method between radar data and visual target detection data, and belongs to the field of video image processing and data fusion. The method comprises the following specific steps: S1, acquiring a video image containing vehicle and pedestrian information by using a vehicle-mounted camera; s2, obtaining an object candidate frame and distance and angle information detected by a radar sensor; s3, extracting an object candidate box for visual target detection from the video image; s4, matching the object candidate frame detected by the radar with the object candidate frame detected by the visual target by using an improved D.IOU algorithm, and screening out a high-quality matching result; s5, on the basis of radar detection information in a high-quality matching result, generating specific regions of interest for different types of vehicles and pedestrians; s6, carrying out secondary matching on the ROI and an object candidate frame of visual target detection, and further screening out a high-quality overlapped part; s7, mapping the distance, angle and other information provided by the radar in the high-quality matching result to a visual target detection frame; and S8, finally outputting a visual target detection frame containing distance and angle information detected by the radar. According to the method, an efficient rapid fusion method between the radar data and the visual target detection data is provided, and the radar data and the visual target detection data can be fused in an extremely short time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video image processing and data fusion, and in particular to a method for fast fusion of radar data and visual target detection data. Background Art

[0002] With the development of self-driving cars, the speed of self-driving cars is getting faster and faster, and the timeliness requirements of target detection during driving are getting higher and higher. The common target detection is to fuse radar data with visual detection data to obtain more accurate target detection results. With the increase in the speed of the car, the speed requirements of target detection are getting higher and higher. Relatively speaking, the speed and accuracy of data fusion should also be improved accordingly.

[0003] At present, the common data fusion in the target detection process usually uses D.IOU or C.IOU to match the selected box. This matching method does not take into account the characteristics of radar data, resulting in unreliable matching results. At the same time, most common matching methods generate ROI first and then match. This method will produce many unnecessary matching attempts, greatly reduce the matching speed, and is not conducive to handling emergencies during high-speed driving of the car.

[0004] In the field of object detection, in order to make up for the shortcomings of the original IOU (Intersection over Union), researchers proposed D.IOU, whose full name is Distance-IoU. It is an extended version of IoU (Intersection over Union) and is mainly used in object detection tasks to improve the regression loss function of the bounding box. IoU is a commonly used indicator to evaluate the accuracy of bounding box predictions, but when two bounding boxes are similar in shape but have a large difference in position, the gradient information of IoU is not sufficient, making model optimization difficult. D.IOU overcomes this problem by introducing information about the distance between the center points of the bounding boxes. The calculation of D.IOU is based on IoU and adds two additional dimensions to consider the positional relationship of the bounding boxes: Where: d 2 represents the square of the Euclidean distance between two matching boxes, c 2 It represents the square of the diagonal length of the minimum enclosing box covering two boxes. However, this method is more suitable for simple target detection, which makes it not work well in the fusion of radar data and visual target detection data.

[0005] In order to solve the above problems, the present invention proposes an improved D.IOU algorithm, which fully considers the characteristics of radar detection data and visual target detection data, breaks through the limitations of traditional methods, and can better match. Summary of the invention

[0006] The present invention provides a method and system for rapid fusion of radar data and visual target detection data. The method combines a variety of technical means and effectively solves the problems of low matching accuracy and slow speed of traditional methods.

[0007] The present invention is implemented by the following technical solution: a method for rapid fusion of radar data and visual target detection data, characterized in that the steps are:

[0008] S1. Use the vehicle-mounted camera to collect video images containing vehicle and pedestrian information;

[0009] S2. Obtain the object candidate frame and distance and angle information detected by the radar sensor;

[0010] S3. Extracting object candidate boxes for visual target detection from video images;

[0011] S4. Use the improved D.IOU algorithm to match the object candidate box detected by the radar with the object candidate box detected by the visual target, and filter out high-quality matching results;

[0012] First, the information obtained from S2 and S3 is used to obtain an improved D.IOU algorithm. Roughly match the two object candidate boxes;

[0013] Among them, α is the optimal weight, its value is about 10, P_imformation ^2 (b,b gt ) is the Euclidean distance between the radar detection point coordinates and the corresponding point coordinates of the visual target object candidate box, and the rest is the same as the original D.IOU;

[0014] For the first matching result, the D.IOU threshold obtained from multiple experiments is used, and the part with a D.IOU value greater than -0.75 is retained and transferred to S5 for processing.

[0015] S5. Generate specific regions of interest (ROIs) for different types of vehicles and pedestrians based on the radar detection information in the high-quality matching results;

[0016] For the same radar detection frame, ROIs of different sizes are generated based on the same radar detection information points;

[0017] S6. performing secondary matching on the ROI and the candidate object box of the visual target detection to further screen out the high-quality overlapping parts;

[0018] S7. Mapping information such as distance and angle provided by the radar in the high-quality matching result to the visual target detection frame;

[0019] S8. The final output is a visual target detection box containing the distance and angle information detected by the radar.

[0020] The present invention provides a radar data and visual target detection data fusion system, which is used to implement a rapid fusion method between the above radar data and visual target detection data. The system includes: a camera, a rough matching module, a data filtering module, a ROI generation module, and a fine matching module;

[0021] The rough matching module uses the improved D.IOU algorithm in S4 to match the radar detection frame and the visual target detection frame;

[0022] The data filtering module calculates the corresponding α weight and threshold through multiple experiments, and selects high-quality matching results based on this. The results of multiple experiments are as follows: Figure 8 shown.

[0023] The ROI generation module generates different ROIs on the original radar detection frame according to the characteristics of different vehicles and pedestrians, as in S5, according to the size of the detected object;

[0024] The camera collects video frames containing vehicle and pedestrian information, obtains the candidate frames of objects detected by the radar, and obtains information such as distance and angle. At the same time, the video images are recognized to obtain the candidate frames of objects for visual target detection. The improved D.IOU algorithm is used to roughly match the two candidate frames, and the part with a D.IOU threshold greater than -0.75 is retained. Then, the ROI generation module generates ROIs of different sizes based on the same base point, and then these ROIs and visual target detection frames are finely matched based on the original D.IOU. Finally, the results of the fine matching are filtered again and output to obtain the visual target detection frame containing the distance and angle information detected by the radar.

[0025] The invention creates beneficial effects: the invention improves the matching accuracy through the above method and the improved matching algorithm, and reduces the total number of matching times through two different matchings, significantly reducing the speed of data fusion, so that the accuracy and timeliness of the entire system are greatly improved. Figure 7 As shown in the figure, the matching algorithm with optimized parameters reduces the time consumption by about 1ms compared to the original brute force matching, and the speed reaches 0.2ms / frame. In addition, the quality of the matched results only decreases slightly, and the overall D.IOU average value of the matching results is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the method of the present invention;

[0027] Figure 2 This is a schematic diagram of vehicle and pedestrian information collected by the camera;

[0028] Figure 3 Schematic diagram for reading radar detection candidate boxes and visual target detection candidate boxes;

[0029] Figure 4 This is a schematic diagram of the ROI generated based on the radar detection selection box;

[0030] Figure 5 This is a schematic diagram of the matching results of the radar detection selection box and the visual target detection selection box;

[0031] Figure 6 This is a schematic diagram of data fusion results;

[0032] Figure 7 Schematic diagram for comparing the performance of the improved method with the original method;

[0033] Figure 8 Schematic diagram of matching performance under different α weights and D.IOU thresholds. DETAILED DESCRIPTION

[0034] Step 1: Video frame acquisition

[0035] Use the camera as a video acquisition module to obtain video frames of pedestrians and vehicles. The collected video data is generally a real-time picture of the vehicle's movement.

[0036] Step 2: Read radar detection data

[0037] Read radar detection data, which includes radar detection selection box and radar detected distance angle and other information.

[0038] Step 3: Read the object selection box for visual target detection in the video frame

[0039] Read the information of the objects to be selected, such as vehicles and pedestrians, identified by the convolutional neural network.

[0040] Step 4: Match the two candidate boxes and keep the high-quality results

[0041] The two candidate boxes are matched through the improved D.IOU algorithm, and the algorithm retains the high-quality results under specific weights and thresholds.

[0042] The improved D.IOU algorithm modifies the original D.IOU according to the format of radar detection data. Perform matching so that the matching results are more suitable;

[0043] The α in the above formula is the optimal weight found after many experiments, and its value is about 10. ^2 (b,b gt) is the Euclidean distance between the coordinates of the radar detection point and the coordinates of the corresponding point of the visual target object candidate box, and the rest is the same as the original D.IOU.

[0044] For the result filtering in step 4, a better D.IOU threshold is obtained through multiple experiments, and only the part with a threshold greater than -0.75 is retained, such as Figure 8 As shown, the second matching number is reduced without reducing performance.

[0045] Step 5: Generate different ROIs

[0046] Based on the original radar detection frame, multiple ROIs of different proportions are generated at the same base point according to the characteristics of vehicles and pedestrians. For the same radar detection frame, ROIs of different sizes are generated based on the same radar detection information points. Generally, the front size area of ​​the vehicle is selected as 2m*2m and 4m*4m (covering cars and trucks), the side size area of ​​the vehicle is selected as 5m*2m and 8m*4m (covering cars and trucks), the pedestrian size area is selected as 0.7m*1.4m, 0.8m*1.6m, 0.9m*1.8m, 1m*2m (adapting to pedestrians of different sizes), and the side size area of ​​non-motor vehicles is selected as 2m*0.8m.

[0047] Step 6: Match and filter again

[0048] The visual target detection matching results obtained in step 4 are matched with the various ROIs obtained in step 5 based on the original D.IOU, and a high-quality matching result with a larger D.IOU value is output.

[0049] Step 7: Assign the distance and angle data detected by the radar to the visual target detection frame

[0050] Assign the angle and distance data detected by the radar to the matched visual target detection data frame.

[0051] Step 8: Output the final result

[0052] Output visual object detection box with radar detection information.

[0053] Embodiment 1:

[0054] In this example, the video image is demonstrated using a road test video. When the system is started, Figure 2 As shown in , it is a video frame containing vehicle and pedestrian information. These video frames will be marked with radar detection frames and visual target detection frames, such as Figure 3 For these data, a rough match is performed first, and then for each radar detection frame of the first matching high-quality result, a ROI with different proportions based on the same base point is generated, such as Figure 4As shown. After performing a second match on these ROIs, Figure 5 As shown. Finally, the visual target detection box with radar data above a specific threshold is output, such as Figure 6 shown.

Claims

1. A fast fusion method between radar data and visual target detection data, It is characterized in that The steps are: S1. Use the vehicle-mounted camera to collect video images containing vehicle and pedestrian information; S2. Obtain the object candidate frame and distance and angle information detected by the radar sensor; S3. Extracting object candidate boxes for visual target detection from video images; S4. Use the improved D.IOU algorithm to match the object candidate box detected by the radar with the object candidate box detected by the visual target, and filter out high-quality matching results; S5. Generate specific regions of interest (ROIs) for different types of vehicles and pedestrians based on the radar detection information in the high-quality matching results; S6. performing secondary matching on the ROI and the candidate object box of the visual target detection to further screen out the high-quality overlapping parts; S7. Mapping information such as distance and angle provided by the radar in the high-quality matching result to the visual target detection frame; S8. The final output is a visual target detection box containing the distance and angle information detected by the radar.

2. A method for rapid fusion of radar data and visual target detection data according to claim 1, It is characterized in that The S4 is the information obtained from S2 and S3, using the improved D.IOU algorithm Roughly match the two object candidate boxes; Among them, α is the optimal weight, P_imformation ^2 (b,b gt ) is the Euclidean distance between the coordinates of the radar detection point and the coordinates of the corresponding point of the visual target object candidate box; For the results of the first match, the D.IOU threshold for the first match screening is obtained based on multiple experiments, and the parts with D.IOU values ​​greater than -0.75 are retained and transferred to S5 for further processing.

3. The method for rapid fusion of radar data and visual target detection data according to claim 1, It is characterized in that The S5 generates ROIs of different sizes for the same radar detection frame based on the same radar detection information points according to the morphological features of different vehicles and pedestrians.

4. The method for rapid fusion of radar data and visual target detection data according to claim 1, It is characterized in that The S6 is to match the ROI generated by S3 with the visual target detection frame again using the original D.IOU, and select the high-quality result part for output.

Citation Information

Patent Citations

  • Data fusion dynamic vehicle detection method based on millimeter wave radar and machine vision

    CN111368706A

  • Traffic scene target detection and positioning method fusing vision and laser radar

    CN114648549A

  • Target detection method based on millimeter wave radar and visual fusion

    CN116385980A

  • Small target detection method and system based on laser radar and camera fusion, and medium

    CN117671635A