Multi-sensor decision level fusion method based on KM matching

Abstract

The invention provides a multi-sensor decision level fusion method based on KM matching, which is specially designed for efficient matching of radar and camera data in an actual road environment. According to the method, on the basis of a traditional serial KM matching algorithm, a boundary frame detected by a radar and a camera is divided into a left side and a right side by presetting a segmentation line and a buffer area, and obstacles intensively distributed on the two sides of a road are independently processed, so that the matching accuracy and efficiency are improved. Meanwhile, a thread pool mechanism and an OpenMP parallel instruction are introduced, multi-thread parallel computing is achieved, the computing process of an IOU matrix, a distance difference matrix and a weight matrix is accelerated, and the matching time is remarkably shortened. In the method, IOU and distance difference value self-defined weight calculation are combined, and virtual nodes are introduced to adapt to the condition that the detection numbers of multiple sensors are different, and the common limitation of a KM algorithm is solved. Compared with a traditional KM algorithm for multi-sensor decision-level fusion, the method adopts the idea of division and treatment. Through region segmentation and parallel calculation, the matching accuracy and the calculation speed are significantly improved, and the stability of the algorithm in a road environment is enhanced. According to the invention, an efficient and reliable solution is provided for environmental perception realized by multi-sensor fusion, and the method is suitable for complex and changeable actual road scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and multi-sensor data fusion, and specifically relates to a multi-sensor decision-level fusion method based on Hungarian matching (KM matching). The method is mainly used for data decision-level fusion in a multi-sensor environment, aiming to improve the matching efficiency and accuracy of radar and camera detection targets. By optimizing the matching algorithm and introducing parallel computing technology, the present invention can achieve efficient and accurate target matching in a complex road environment, providing reliable technical support for environmental perception of multi-sensor fusion. Background Art

[0002] With the rapid development of intelligent technology, multi-sensor fusion technology has been widely used in the field of environmental perception, especially in autonomous driving systems. Multi-sensor fusion can provide more comprehensive, accurate and reliable environmental information by integrating data from multiple sensors such as radar, camera, lidar, etc., thereby significantly improving the safety and navigation capabilities of autonomous driving vehicles. In autonomous driving, multi-sensor fusion can not only achieve high-precision detection and positioning of targets such as roads, pedestrians, vehicles and traffic signs, but also enhance the system's anti-interference ability and robustness, ensuring stable operation of vehicles in complex and changing road environments.

[0003] In the process of implementing multi-sensor fusion, commonly used algorithms include fusion methods based on data level, feature level and decision level. Among them, Hungarian Matching (KM matching) is widely used in decision-level fusion methods because it can accurately perform maximum weight matching of bipartite graphs in multi-sensor data. The main advantage of KM matching is that it can handle the matching relationship between a large number of targets according to the weights between targets while ensuring matching accuracy. In addition, by constructing a weight matrix, KM matching can comprehensively consider the confidence and matching conditions of the targets detected by each sensor to improve the overall matching effect. However, traditional KM matching has certain limitations in practical applications. First, as a classic bipartite graph matching algorithm, KM matching fails to fully consider the spatial distribution characteristics of targets in a specific environment. For example, in a road environment, obstacles are usually mainly distributed on the left and right sides of the road, while the middle area is relatively empty. Traditional KM matching lacks the use of this spatial characteristic, resulting in a significant increase in computational complexity when processing a large number of targets, and matching efficiency and real-time performance are limited. Secondly, when dealing with an unbalanced number of targets detected by sensors, traditional KM matching is prone to matching deviations, which affects the overall matching effect. In addition, KM matching usually runs in a serial manner, which makes it difficult to fully utilize the parallel computing capabilities of modern multi-core processors, further limiting its application in scenarios with high real-time requirements.

[0004] In view of the above problems, the present invention proposes a multi-sensor decision-level fusion method based on KM matching. Considering that in actual traffic conditions, obstacles are usually divided into the left and right sides of themselves by traffic lines, if all target information is matched, matching the targets distributed on the left and right sides will lead to a waste of resources. Therefore, the present invention introduces a dividing line based on road characteristics, divides the bounding box detected by the sensor into the left and right sides according to the spatial distribution, and optimizes the matching process in combination with the area distribution. At the same time, multi-threading technology and thread pool management are adopted to realize efficient parallel calculation of IOU matrix and distance difference matrix, significantly improving the running speed and real-time performance of the matching algorithm. In addition, considering that the number of targets detected by multiple sensors may be unbalanced, resulting in the inability to construct the weight matrix normally, the present invention introduces virtual nodes to balance the inconsistency of the number of targets detected by different sensors, ensuring that KM matching can still efficiently and accurately perform target matching in unbalanced matching scenarios. Through these improvements, the present invention comprehensively improves the matching efficiency and accuracy of multi-sensor data fusion, and meets the strict requirements of high real-time applications such as autonomous driving. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-sensor decision-level fusion method based on KM matching, which is specifically used to achieve efficient and accurate matching of radar and camera data in a road environment. Based on the traditional KM matching algorithm, this method makes full use of multi-threading and parallel computing technology, significantly improves the running speed and matching accuracy of the matching algorithm, and solves the problems of low efficiency and inaccurate matching in the multi-sensor data fusion process in the KM matching algorithm. In addition, the present invention introduces a virtual node mechanism to handle the imbalance in the number of radar and camera detection targets, thereby ensuring the efficiency and robustness of the KM algorithm in various matching scenarios.

[0006] The design method of multi-sensor decision-level fusion based on KM matching of the present invention includes the boundary box segmentation of the single sensor recognition result, multi-threaded calculation of the intersection of union (IOU) matrix and distance difference matrix of the left and right data after segmentation, the construction of the weight matrix by integrating the IOU matrix and the center distance difference matrix, parallel matching of the left and right sides based on their respective weight matrices, and integration of matching results, including the following steps:

[0007] Step 1. Obtain the result of single sensor target identification, and segment the bounding box based on the preset segmentation line. Preprocess the point cloud data from the radar and the image data from the camera to obtain the point cloud Euclidean clustering result. After coordinate transformation, obtain the two-dimensional obstacle box information. At the same time, use YOLOv5s to recognize the image and obtain the two-dimensional obstacle box. Use the point cloud processing results and YOLOv5s recognition results as the input of the entire algorithm. Based on the preset segmentation line and buffer zone, segment the bounding boxes (Bounding Box) detected by the preprocessed radar and camera, divide the two types of bounding boxes into left and right sides, and add the bounding boxes on the same side after segmentation to the same matching class;

[0008] Step 2: Use multi-threading technology and thread pool management to parallelize the left and right data after segmentation. Specifically, it includes calculating the intersection over union (IOU) matrix between the radar and camera bounding boxes, and calculating the distance difference matrix between the center points of the radar and camera bounding boxes. By using OpenMP and thread pool technology, efficient parallel calculation of the IOU matrix and distance difference matrix is ​​achieved;

[0009] Step 3. Combine the calculated IOU matrix and distance difference matrix to construct a weight matrix. For each pair of radar bounding box and camera bounding box, if the IOU value is less than the preset threshold (IOU_THRESHOLD) or the distance difference is greater than the preset threshold (DISTANCE_THRESHOLD), the matching weight of the pair is set to negative infinity to avoid inappropriate matching; otherwise, the weighted weight is calculated according to the IOU value and the distance difference. The weight calculation formula is: weight = α*IOU*IOU_SCALE+(1-(distance difference / DISTANCE_THRESHOLD))*DISTANCE_SCALE, where α is the weight factor in the weight calculation;

[0010] Step 4: Run KM matching in parallel based on the weight matrices of the left and right bounding box sets. Initialize the labels and matching relationships, use the augmented path search and label update strategy based on breadth-first search (bfs) to accelerate the matching process, and use the weight threshold to filter out inappropriate matching pairs to ensure high accuracy and efficiency of the matching results. Process multiple matching pairs in parallel. In addition, when constructing the matching matrix, introduce virtual nodes to balance the number of targets detected by the radar and camera to ensure that the KM algorithm can still run efficiently in matching scenarios with an unbalanced number of targets.

[0011] Step 5: Integrate the matching results and unmatched radar and camera targets on the left and right sides. Specifically, it includes collecting the matching pairs on the left and right sides and converting them into raw data indexes; merging unmatched radar targets and unmatched camera targets, and storing them as unmatched radar target lists and unmatched camera target lists respectively. For the successfully matched targets, they are used as the final target information; for the unmatched targets, they are directly used as independent target information.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] 1. High matching accuracy: By dividing the bounding box into left and right parts based on the preset segmentation line and buffer zone, obstacles concentrated on both sides of the road are processed independently, reducing interference between different areas and significantly improving matching accuracy;

[0014] 2. Significant improvement in computing speed: Multi-threading technology and thread pool management are used, combined with OpenMP parallel instructions to achieve efficient parallel computing of IOU matrix and distance difference matrix, which greatly shortens the execution time of the matching algorithm and meets real-time processing requirements;

[0015] 3. Enhanced algorithm robustness: By introducing a virtual node mechanism, the imbalance in the number of targets detected by radar and camera is handled, ensuring the efficiency and robustness of KM matching in various matching scenarios and avoiding the occurrence of false matching; BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 System architecture diagram of multi-sensor decision-level fusion method based on KM matching

[0017] Figure 2 Flowchart of bounding box segmentation for multi-sensor decision-level fusion method based on KM matching

[0018] Figure 3 Constructing a flow chart for the weight matrix of the multi-sensor decision-level fusion method based on KM matching

[0019] Figure 4 The matching flow chart of the multi-sensor decision-level fusion method based on KM matching

[0020] Figure 5 Comparison of average running time of KM matching before and after optimization DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0022] The system architecture of the multi-sensor decision-level fusion method based on KM matching is shown in the figure below. Figure 1As shown, it includes the following steps:

[0023] Step 1. After obtaining the radar 2D bounding box set radar_boxes and the camera 2D bounding box set camera_boxes obtained by performing target recognition, considering the actual road conditions, set 1 / 2 of the image length as the position of the split line SPLIT_X to divide the image into left and right parts. Preset SPLIT_BUFFER as the buffer width to handle the situation where the bounding box is close to the split line. For a bounding box A1 with the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) on the image, calculate the horizontal coordinate of its center point center_x = (x1 + x2) / 2. According to the relationship between center_x, SPLIT_X and SPLIT_BUFFER, allocate it to the left or right side according to the following strategy:

[0024] If center_x < (SPLIT_X - SPLIT_BUFFER), the bounding box is completely on the left;

[0025] If center_x > (SPLIT_X - SPLIT_BUFFER), the bounding box is completely on the right side;

[0026] If (SPLIT_X-SPLIT_BUFFER)≤center_x≤(SPLIT_X+SPLIT_BUFFER), the bounding box is located in the middle buffer. At this time, the bounding box is allocated to the left or right side according to the overlapping area with the split line, and its area on the left and right sides of the split line is calculated;

[0027] The left area area_left = (SPLIT_X-x1) × (y2-y1), the right area area_right = (x2-SPLIT_X) × (y2-y1), if the left area area_left > the right area area_right, split it into the left match, otherwise split it into the right match;

[0028] According to the above strategy, the final output results are radar_left_boxes, radar_right_boxes and camera_left_boxes, camera_right_boxes. The program flow chart of segmentation boundary box is as follows: Figure 2 As shown;

[0029] Step 2. After completing the segmentation of the bounding box, the system needs to calculate the intersection-over-union (IOU) matrix and distance difference matrix between the radar and camera bounding boxes for the left and right sides respectively. First, create the Matcher classes for the left and right sides, add radar_left_boxes and camera_left_boxes to one class, and radar_right_boxes and camera_right_boxes to another class. Then use the thread pool ThreadPool to submit the left and right matching tasks in parallel, and calculate the IOU matrix and distance difference matrix respectively.

[0030] For each pair of radar bounding box B_radar_i and camera bounding box B_camera_j, calculate their intersection area and union area, and then calculate IOU. The calculation method is as follows:

[0031]

[0032] Where IntersectIon Area is the area where the radar bounding box and the camera bounding box overlap, and UnionArea is the area outside the overlapping area of ​​the radar bounding box and the camera bounding box. If the two bounding boxes do not overlap, IOU_{i,j}=0;

[0033] For the calculation of the distance difference matrix, the Euclidean distance between the center points of each pair of radar and camera bounding boxes is calculated as follows:

[0034]

[0035] For non-existent bounding boxes, such as virtual nodes, the center distance is the maximum value VIRTUAL_DISTANCE to avoid unnecessary pairing;

[0036] Step 3: After calculating the IOU matrix and the distance difference matrix, the system combines the two matrices to construct a weight matrix to evaluate the matching priority of each pair of radar and camera bounding boxes. The required preset parameters of the weight matrix are shown in Table 1 below:

[0037]

[0038] Table 1 Preset parameter names and values

[0039] The construction of the weight matrix based on the parameters in the above table is based on the following strategy:

[0040] For each pair of radar bounding box B_radar_i and camera bounding box B_camera_j, the weight is calculated according to the following conditions:

[0041] If IOU_{i,j} < IOU_THRESHOLD, i.e., the intersection over union is lower than the threshold, or distance_diff_{i,j} >= DISTANCE_THRESHOLD, i.e., the center point distance exceeds the threshold, or distance_diff_{i,j} == VIRTUAL_DISTANCE, i.e., it involves virtual nodes, if any of the above conditions is met, set the weight WEIGHT_THRESHOLD to -1000, and it will not be considered for matching in the subsequent process. Except for the above cases, calculate the weight weight_{i,j} according to the following formula:

[0042]

[0043] Traverse the IOU matrix and the distance difference matrix, calculate the weights of each pair of bounding boxes according to the above conditions, and fill the weight matrix weight. For the weights between virtual nodes, directly set them to WEIGHT_THRESHOLD to avoid participating in the matching. Finally, obtain a weight matrix of size N×N, where N is the larger value of the radar and camera bounding boxes. Taking the left-side data processing as an example, as Figure 3 is the flowchart for constructing the left-side weight matrix;

[0044] Step 4: After obtaining the constructed weight matrix, the system matches the radar and camera bounding boxes on the left and right sides respectively. Using the thread pool ThreadPool, submit the matching tasks on the left and right sides in parallel and execute the KM algorithm respectively;

[0045] For each weight matrix, initialize the labelU array to the maximum weight of each row, initialize the labelV array to 0, and then use the breadth-first search strategy to improve the speed of finding the augmenting path. During the search process, the algorithm marks the visited radar and camera bounding boxes and records the path through the parent node array. When an augmenting path is found, update the matching result, match the corresponding radar and camera bounding boxes, adjust the label values, and search for the next augmenting path;

[0046] Repeat the execution of the augmenting path search and label adjustment until all possible matching pairs are processed to ensure the global optimality of the matching result, as Figure 4 is the matching flowchart for KM matching. Then obtain three sets, namely final_matches, the pairs of radar and camera bounding boxes with successful matching; unmatched_radar, the set of indices of unmatched radar bounding boxes; unmatched_camera, the set of indices of unmatched camera bounding boxes;

[0047] Step 5: Integrate the matching results. After the matching tasks on the left and right sides are completed, the system integrates the matching results into the final set of matching pairs and performs different processing on each pair, as follows:

[0048] For the bounding boxes of the successfully matched camera and radar, the overlapping area of ​​the two bounding boxes is used as the final target bounding box, and the confidence and category information provided by the camera yolov5 after recognition and the spatial distance information provided by the radar are retained as the attributes of the bounding box to achieve the fusion of multi-sensor information;

[0049] For unmatched camera bounding boxes, the radar may not recognize the target due to factors such as the physical rebound of the emitted point cloud. In this case, the confidence and category information after yolov5 recognition are used as its attributes to avoid the target loss caused by the laser radar failing to recognize the target.

[0050] For unmatched radar bounding boxes, the camera may fail to successfully identify them due to factors such as lighting, so the spatial distance of the bounding box identified by the radar and the category information misc (location category) are retained as its final attributes;

[0051] In order to verify the improvement of the processing efficiency of the algorithm proposed in this patent compared with the original algorithm, ten comparative experiments were conducted under the same experimental environment using 447 frames of images and point cloud data collected on September 26, 2011 provided by the Kitti dataset. Figure 5 The data shown is the time it takes for the matching algorithm to complete the matching of each frame of data. It can be seen that the optimized algorithm has a significant improvement in processing speed.

Claims

1. A multi-sensor decision-level fusion method based on KM matching, characterized in that: The following steps are involved: Step 1. Obtain the result of single sensor target identification, and segment the bounding box based on the preset segmentation line. Preprocess the point cloud data from the radar and the image data from the camera to obtain the point cloud Euclidean clustering result. After coordinate transformation, obtain the two-dimensional obstacle box information. At the same time, use YOLOv5s to recognize the image and obtain the two-dimensional obstacle box. Use the point cloud processing results and YOLOv5s recognition results as the input of the entire algorithm. Based on the preset segmentation line and buffer zone, segment the bounding boxes (Bounding Box) detected by the preprocessed radar and camera, divide the two types of bounding boxes into left and right sides, and add the bounding boxes on the same side after segmentation to the same matching class; Step 2: Use multi-threading technology and thread pool management to parallelize the left and right data after segmentation. Specifically, it includes calculating the intersection over union (IOU) matrix between the radar and camera bounding boxes, and calculating the distance difference matrix between the center points of the radar and camera bounding boxes. By using OpenMP and thread pool technology, efficient parallel calculation of the IOU matrix and distance difference matrix is ​​achieved; Step 3. Combine the calculated IOU matrix and distance difference matrix to construct a weight matrix. For each pair of radar bounding box and camera bounding box, if the IOU value is less than the preset threshold (IOU_THRESHOLD) or the distance difference is greater than the preset threshold (DISTANCE_THRESHOLD), the matching weight of the pair is set to negative infinity to avoid inappropriate matching; otherwise, the weighted weight is calculated according to the IOU value and the distance difference. The weight calculation formula is: weight = α*IOU*IOU_SCALE+(1-(distance difference / DISTANCE_THRESHOLD))*DISTANCE_SCALE, where α is the weight factor in the weight calculation; Step 4: Run KM matching in parallel based on the weight matrices of the left and right bounding box sets. Initialize the labels and matching relationships, use the augmented path search and label update strategy based on breadth-first search (bfs) to accelerate the matching process, and use the weight threshold to filter out inappropriate matching pairs to ensure high accuracy and efficiency of the matching results. Process multiple matching pairs in parallel. In addition, when constructing the matching matrix, introduce virtual nodes to balance the number of targets detected by the radar and camera to ensure that the KM algorithm can still run efficiently in matching scenarios with an unbalanced number of targets. Step 5: Integrate the successfully matched results and the unmatched radar and camera targets on the left and right sides. Specifically, it includes collecting the matching pairs on the left and right sides and converting them into raw data indexes; merging the unmatched radar targets and the unmatched camera targets, and storing them as the unmatched radar target list and the unmatched camera target list respectively. For the successfully matched targets, use them as the final target information; For targets that are not successfully matched, they are directly treated as independent target information.

2. According to claim 1, the result of obtaining the target identification by a single sensor is obtained, and the boundary box is segmented based on a preset segmentation line, characterized in that: The following steps are included: Step 1: Set the split line and buffer according to the image resolution, select the midpoint of the image in the horizontal direction as the split line position, set the split line position to SPLIT_X=621.0 and the buffer width to 50.0 according to the image resolution; Step 2: For each bounding box, calculate the coordinates of its center point to obtain the spatial distribution information of the bounding box. Determine whether the x coordinate of the center point of the bounding box is within the range of [571.0, 671.0]. If not, determine which side the bounding box is divided to based on the x value of the center point coordinate. When x < 571.0, it is divided to the left side, and when x > 671.0, it is divided to the right side. Step 3. If the x-coordinate of the center point of the bounding box is in [571.0, 671.0], compare its area on the left and right sides of the dividing line. If the area on the left is larger than that on the right, divide the bounding box to the left. If the area on the right is larger than that on the left, divide the bounding box to the right.

3. According to claim 1, the method of using multi-threading technology and thread pool management to parallelly calculate the split left and right side data is characterized in that: The bounding box is divided to construct two completely independent matching events, and the thread pool and OpenMP technology are used to achieve efficient parallel processing of the left and right data after segmentation. Specifically, it includes: Initialize a thread pool with the number of threads equal to the number of threads supported by the hardware, or use the preset number of threads by default when the detection cannot be performed. Manage multi-threaded tasks through the thread pool to avoid frequent creation and destruction of threads, and improve system stability and efficiency; The calculation of the IOU matrix and the distance difference matrix are assigned to different threads respectively, and these calculation tasks are executed in parallel, using the parallel for instruction of OpenMP and the task scheduling mechanism of the thread pool; Synchronization and resource management: In the parallel computing process, the mutual exclusion lock (mutex) and condition variable (conditionvariable) synchronization mechanism are used to perform safe resource access and task coordination in a multi-threaded environment.

4. The method of claim 1, wherein the KM matching is performed in parallel based on the weight matrices of the left and right bounding box sets, wherein: Specifically include: Virtual nodes are introduced. Determine the difference between the number of radar detected targets and the number of camera detected targets, and introduce virtual nodes to the side with fewer targets so that the number of radar and camera detected targets is equal. Virtual nodes are used to represent the situation where there is no actual matching object; Weight setting. For the matching between virtual nodes and actual nodes, set the weight to a preset value (such as negative infinity) to avoid mismatching between actual nodes and virtual nodes. Virtual nodes are only used to balance the dimensions of the matching matrix and do not participate in the actual target matching; During the operation of the KM algorithm, the introduction of virtual nodes ensures that each actual node has a corresponding matching object (including virtual nodes), thereby ensuring the integrity and consistency of the matching process. In the matching results, the matching of the actual node and the virtual node indicates that the node has not found a suitable matching object and should be regarded as an unmatched target.

Images