A target localization method based on fusion of visual image and sparse point cloud
Through the fusion method of sparse point clouds and visual images, combined with YOLOv3, KD-Tree, DBSCAN and visual features, the complexity and slow speed of the existing target positioning methods are solved, and high-precision and low-cost automatic driving target positioning is achieved.
Patent Information
- Application Number
- CN202210669075.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-14
- Publication Date
- 2025-08-12
- Estimated Expiration
- 2042-06-14
AI Technical Summary
The existing target positioning methods rely heavily on high-wire harness laser point clouds, resulting in complex algorithms, slow detection speed and high cost, making it difficult to meet the needs of commercial implementation of autonomous driving technology.
The method of post-fusion of sparse point clouds and visual images is adopted, and 2D detection information is obtained through the end-to-end YOLOv3 object detection algorithm, 3D position information is obtained by combining point cloud instance segmentation and unsupervised clustering methods, and target-level fusion is performed through visual projection, point cloud segmentation and clustering is used using KD-Tree and DBSCAN, and visual features are added for multi-object tracking to improve accuracy and continuity.
In actual traffic scenarios, the detection accuracy of different categories of targets is more than 90% and the positioning error is less than 0.1m, meeting the real-time and accuracy requirements of autonomous driving and reducing costs.
Smart Images
Figure CN115187955B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving environment perception technology, and in particular to a target positioning method based on point cloud and image fusion. Background Art
[0002] Environmental perception is key to autonomous driving's interaction with the external environment. Its core lies in understanding the driving dynamics of both the vehicle and its surroundings. In urban traffic scenarios, with numerous participants and complex and ever-changing traffic conditions, the positioning of road targets is crucial and a prerequisite for autonomous driving to perform path planning, decision-making, and safe driving.
[0003] Currently, most target localization methods rely heavily on high-beam laser point clouds, which suffer from common issues such as complex algorithms, slow detection speeds, and high costs, making them difficult to meet the needs of commercial autonomous driving technology. Methods based on the fusion of laser point clouds and vision often overcome the shortcomings of single sensors and meet the real-time requirements of autonomous driving perception systems, thus having practical significance for the commercialization of autonomous driving. Summary of the Invention
[0004] Considering that most current target positioning methods heavily rely on dense point clouds, which have the problems of large computational complexity, slow detection speed and high cost, the present invention draws on the idea of multi-sensor fusion, and based on the fusion of sparse point clouds (16-beam lidar point clouds) and visual images to fully exploit the features of point clouds and images, and proposes an autonomous driving fusion positioning method with high detection accuracy, fast speed and low cost.
[0005] Specifically, image detection uses the end-to-end object detection network YOLOv3 to obtain 2D detection information; point cloud detection uses point cloud instance segmentation and unsupervised clustering to obtain 3D object position information; and target localization is achieved by fusing 2D detection information with 3D position information after visual projection. This method achieves detection accuracy exceeding 90% for various target categories in real-world traffic scenarios, with a localization error of 0.1m, effectively meeting the requirements of autonomous driving target localization.
[0006] This paper mainly addresses the common problems of most current target positioning methods that rely heavily on high-beam laser point clouds, such as complex algorithms, slow detection speed, and high costs. It proposes a target positioning method based on the fusion of visual images and sparse point clouds. The method includes:
[0007] Step 1: Use a camera to obtain real-time 2D image data;
[0008] Step 2: Use LiDAR to obtain real-time 3D scanning point cloud data;
[0009] Step 3: The 2D image data obtained in step 1 and the 3D point cloud data obtained in step 2 are synchronized in time according to the timestamps, and spatial synchronization is performed using a joint calibration method;
[0010] Step 4: Perform target detection on the 2D image data obtained in step 1 to obtain 2D image detection results, exclude small objects in the distance based on the pixel size of the detection frame; and use the SORT tracking method to perform multi-target tracking based on the detection results.
[0011] Step 5: Divide the 3D scanning point cloud data obtained in step 2 into multiple sectors according to their spatial positions, with the scanning point as the center, and divide the point cloud data into sector rings within each sector;
[0012] The sector ring segmentation method is: set the distance parameter and When each point satisfies formula (1), it is considered that the point is in the jth sector ring.
[0013]
[0014] Among them, (x i +y i ) is the projection coordinate of the i-th data point in the XY plane;
[0015] Step 6: Convert the Cartesian coordinate system into a form similar to the polar coordinate system and perform straight line fitting on the dz plane;
[0016]
[0017] Where θ represents the angle between the laser line where the data point is located and the Z axis;
[0018] Set the slope threshold of the laser line where the data point is located, the angle threshold between the laser line where the data point is located and the Z axis, and the straight line fitting error threshold of the laser line where the data point is located. Based on these thresholds, determine whether the data point is ground point cloud data and remove the ground point cloud data;
[0019] Step 7: Segment the remaining point cloud data and cluster the segmented point cloud data to obtain the target's 3D information including position, size, etc.
[0020] Step 8: Associate the 2D detection results identified in step 4 with the 3D targets clustered in step 7. Map each target obtained in step 7 to the image using the visual projection matrix obtained in step 3. If the projection point falls within the 2D detection box, associate the 2D detection result with the 3D clustering result. Otherwise, the association fails.
[0021] Furthermore, in step 4, the multi-target tracking method SORT is improved. In addition to considering IOU during matching, target features are added, and whether they are the same target is determined by calculating the similarity of target features and IOU.
[0022] Furthermore, the specific method of step 7 is:
[0023] Step 7.1: Create a KD-Tree, loop through each dimension of the 3D point cloud and select the median of the 3D point cloud in that dimension as the splitting hyperplane. Attach the 3D point cloud to the left of the median to the left subtree, and the 3D point cloud to the right of the median to the right subtree. Recursively process the subtrees until all point clouds are organized.
[0024] Step 7.2: Access the data from the root node top to bottom, calculate the distance between the data and the leaf nodes, where the data point corresponding to the minimum distance Dis is recorded as Nearest, perform a backtracking operation to find the nearest neighbor point closer to the data, determine whether there is a point closer to the data in the unvisited branch, update the nearest neighbor point Nearest and the minimum distance Dis, specify the minimum and maximum search thresholds during the search process, abandon the segmentation of the point cloud type when the search result is less than the minimum threshold, and use the maximum search threshold for constraints when there are too many search results to complete the discrete point cloud instance segmentation;
[0025] Step 7.3: Use the DBSCAN method to cluster the segmented point cloud data.
[0026] Furthermore, in step 8, the target point cloud obtained by clustering is projected onto the image through vision. If 60% of the laser points fall within the 2D image detection frame, the association is successful, otherwise the association fails.
[0027] The present invention uses a combination of 2D images and 3D point cloud data to identify targets, combining the advantages of fast 2D speed and high 3D accuracy. The 2D recognition results are used as a reference for 3D recognition, which speeds up the 3D recognition speed. Finally, the 2D results are used for verification, giving the present invention the advantages of fast recognition speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Target localization method process based on fusion of vision and sparse point cloud
[0029] Figure 2 Schematic diagram of ordering of spatial discrete point cloud
[0030] Figure 3 Schematic diagram of the point cloud search process
[0031] Figure 4 Schematic diagram of the matching process based on visual features
[0032] Figure 5 Point cloud visual projection process
[0033] Figure 6 Fusion positioning errors and their distributions of three types of targets at a distance of 20m: (a) pedestrians; (b) electric vehicles; (c) cars. DETAILED DESCRIPTION
[0034] Image 2D detection.
[0035] Considering the real-time requirements of autonomous driving, this paper uses the end-to-end YOLOv3 object detection algorithm. This method abandons the concept of region proposals, significantly improving detection speed while maintaining minimal loss of accuracy. During YOLOv3 post-processing, the pixel size of the detection box is used to eliminate interference from small, distant objects. Detection results are filtered based on the target category in the actual scene to eliminate the impact of false detections on the fusion positioning results.
[0036] Ground filtering of point clouds with different slopes.
[0037] Based on the laser radar scanning method, this paper proposes a method of segmenting the ground point cloud by linear fitting in the vertical plane, which can well solve the problem caused by the change of ground slope. First, we divide the circular surface into 2000 sectors based on the angle of 0.18° between the laser radar beams, and set the distance parameter in each sector. and Divide the part into multiple sectors, such as Figure 2 As shown;
[0038] When each point satisfies formula (1), it is considered that the point is in the jth sector ring, that is, the projection box of the spatial discrete point on the XY plane can be used to determine the box, thereby realizing the ordering of the discrete point cloud;
[0039]
[0040] Secondly, we transform the XY Cartesian coordinate system into a form similar to the polar coordinate system according to formula (2) and perform linear fitting in the dz plane;
[0041]
[0042] Finally, we set a threshold based on the slope of the line, the distance of the line on the Z axis, the line fitting error, the distance from the first point in the box to the line, etc. to determine whether it is a ground point cloud.
[0043] Instance segmentation of spatially discrete point clouds
[0044] In order to maximize the distinction of target point clouds, this patent uses a point cloud instance segmentation method based on KD-Tree to distinguish target point clouds in complex point cloud scenes. The specific segmentation process includes KD-Tree construction and nearest neighbor search based on KD-Tree.
[0045] KD-Tree construction. Each dimension of the 3D point cloud is selected as the splitting dimension in a loop. The median of the 3D point cloud in that dimension is used as the splitting hyperplane. The 3D point cloud to the left of the median is mounted in the left subtree, and the 3D point cloud to the right of the median is mounted in the right subtree. The subtrees are processed recursively until all point clouds are organized.
[0046] Nearest neighbor search of KD-Tree. The distances from all leaf nodes to the root node of a balanced KD-Tree are approximately equal, so it is optimal to use the nearest neighbor search method when searching a balanced KD-Tree. Access the data from the root node top to bottom, and calculate the distance between the data and the leaf nodes. The data point corresponding to the minimum distance Dis is recorded as Nearest. Perform a backtracking operation. The main purpose of backtracking is to find the nearest neighbor point closer to the data, determine whether there is a point closer to the data in the unvisited branch, and update the nearest neighbor point Nearest and the minimum distance Dis. The search process is as follows: Figure 3 As shown in the figure, the minimum and maximum search thresholds need to be specified during the search process. When the search result is less than the minimum threshold, the segmentation of this type of point cloud is abandoned. When the search results are too many, the maximum search threshold is used for constraint, thereby completing the discrete point cloud instance segmentation process.
[0047] Unsupervised clustering of spatially discrete point clouds
[0048] After instance segmentation, the point cloud needs to be clustered to further derive the target's location and size information. The segmented point cloud has significant inter-class spacing, and the laser point cloud density is evenly distributed with some differentiation in distance. Taking these factors into account, this patent uses the DBSCAN method to classify spatially discrete point clouds. DBSCAN is a density-based spatial clustering method that maintains a relatively stable classification effect even in spatial samples with a certain amount of noise.
[0049] Multi-target tracking method based on fusion of visual features
[0050] Image detection is easily affected by light interference, resulting in missed detections and false detections, which leads to discontinuous positioning problems in the target fusion positioning results. The present invention adopts a multi-target tracking method to supervise the target detection output with high-confidence tracking results when the detection is unstable, thereby improving the continuity of fusion positioning. The tracking results of traditional multi-target tracking methods are not satisfactory. The current mainstream SORT tracking method matches targets in upper and lower frame images based on the intersection of detection frames. When occlusion occurs for a long time, the tracking ID is prone to frequent changes. In order to solve this problem, improvements are made here on the basis of the SORT tracking method. Visual features are added when matching targets, and deeper image features are used as a basis. This method has better tracking effects in actual traffic scenarios.
[0051] Specifically, when performing target matching, the idea of calculating the intersection-over-union ratio of the detection frame in the SORT method is still retained. In addition, high-dimensional visual features are used as additional matching basis. The detection frame is mapped to the Feature Map to find the corresponding image features, and a fixed-length feature vector is output through ROI Pooling. It is input to the fully connected layer to obtain the target high-dimensional features. The Euclidean distance is calculated based on this high-dimensional feature vector, and the targets corresponding to the two feature vectors with the smallest distance are determined to be the same target. The matching process based on visual features is as follows Figure 4 shown.
[0052] The vision-based multi-target tracking method proposed in this paper still adopts the concept of current mainstream tracking methods: Detection+Track. This method uses detection results (BBOX, confidence, and target features) as algorithm inputs. Confidence is primarily used to screen detected targets, while BBOX and target features serve as matching criteria. During tracking, motion estimation and data association are performed based on pairwise matching of the visual features of the tracked targets with the position and size of the detection boxes. This increases the likelihood of re-finding the target if lost, solving the problem of frequent ID changes of tracked targets.
[0053] Fusion positioning method based on fusion of visual projection
[0054] Based on the image detection results, multi-target tracking results based on visual features, and laser point cloud segmentation and clustering results, the present invention uses a method based on visual projection to match the point cloud space and image space, and uses a target fusion method to complement and verify the two detection results, thereby obtaining a more reliable 3D target detection output. The visual projection process is as follows Figure 5 shown.
[0055] Considering the advantages of images for classification and point clouds for position regression, a perceptual localization method with high detection accuracy, high speed, and low cost is proposed based on the concept of post-fusion of point clouds and images. For images, object detection methods are used to obtain 2D information of the target. For point clouds, ground filtering, instance segmentation, and unsupervised clustering are used to determine the target's position and size. The detection data is then fused at the object level using visual projection.
[0056] S1. Time synchronization of sparse point cloud and image data. Point cloud and image data are derived from heterogeneous sensors. To obtain data at the same moment, a timestamp alignment method is used to solve the time synchronization problem.
[0057] S2. Spatial synchronization of the sparse point cloud and image data. This is done simultaneously with the temporal synchronization in step S1. Spatial synchronization ensures that the LiDAR and camera have the same receptive field, establishing a correspondence between the point cloud and image data. A joint calibration method is used here to obtain the visual projection matrix.
[0058] S3. 2D object detection in the image. Considering the real-time requirements of autonomous driving, this paper uses the end-to-end YOLOv3 object detection algorithm to process the image based on steps S1 and S2, obtaining 2D object detection information. During YOLOv3 post-detection processing, the pixel size of the detection box and the detection category are used to eliminate false detections to a certain extent.
[0059] S4. Multi-target tracking based on visual features. The matching basis of the upper and lower frame targets in the Sort target tracking method is the intersection-over-union ratio of the detection frame. When the target is occluded for a short time, it is easy to lose the target, which causes the problem of frequent changes in the tracking ID. The present invention considers adding the visual features of the target as an additional matching basis in addition to the intersection-over-union ratio, and the visual features used for matching do not need to be repeatedly extracted. The pixel coordinates of the detection frame obtained according to step S3 can be obtained in the detection network and will not bring too much computation.
[0060] S5. Filtering ground point clouds with different slopes. Based on the point cloud obtained in steps S1 and S2, convert the xy coordinates into polar coordinates. This converts the point cloud {x, y, z} information into {d, z} + θ information. Ray fitting is then performed on the dz plane. Rays close to the ground are screened based on the slope, intercept, and fitting error. The projection of the distance from the point to the ray on the z-axis is used to determine whether it is a ground point cloud.
[0061] S6. Instance segmentation of spatial 3D point clouds. After filtering the ground point cloud in step S5, in order to maximize the distinction between the target point cloud and distinguish it from the complex background, the present invention uses a KD-Tree to organize the spatial point cloud and reduce the dimensionality of the point cloud to reduce the time consumed by the nearest neighbor search method and improve the instance segmentation speed.
[0062] S7. Unsupervised Point Cloud Clustering. For further processing of the point cloud after instance segmentation in step S6, the present invention uses the DBSCAN method to cluster discrete, unordered point clouds in three-dimensional space to obtain the three-dimensional position and size information of the target. As an unsupervised density clustering method, the DBSCAN method does not require a predetermined number of clusters and is insensitive to outliers, making it applicable to point cloud clusters of any shape.
[0063] S8. Fusion localization based on visual projection. This section performs object-level post-fusion on the 2D information detected in step S3 and the 3D information detected based on the point cloud in step S7. Visual projection is used to associate the target information. Specifically, the target position detected by the point cloud is projected onto the image through visual projection. If it is within the detection box, the detection information is associated. If fusion localization fails, the tracking result with the highest confidence is used as the fusion localization result for the current frame.
[0064] Fusion localization experiment
[0065] In order to verify the perceptual positioning effect of sparse point cloud and image fusion, the present invention conducts four positioning experiments on three types of targets, pedestrians, electric vehicles, and cars, 20 meters in front of the car body respectively, and mainly analyzes the accuracy and positioning error of the fusion positioning method.
[0066] In this fusion positioning experiment, pedestrians moved at 5 km / h, electric vehicles at 15 km / h, and cars at 20 km / h in a fixed direction to simulate real-world scenarios. Because target position changes are minimal between frames, downsampling was used to uniformly select 300 frames for statistical analysis. The fusion positioning accuracy test results are shown in Table 1.
[0067] Table 1 Fusion positioning accuracy test results
[0068]
[0069] From the data in the analysis table, we can find that the detection accuracy of electric vehicles and cars at a distance of 20m is maintained above 0.95. Pedestrians are smaller in size, and the number of point clouds hitting the target at 20m is small, which makes it easy to miss detection. However, the detection accuracy is still maintained at around 0.9. This shows that the fusion positioning method has good detection accuracy.
[0070] err D =abs(DD truth ) (3)
[0071] When conducting the error analysis experiment of the fusion positioning method, we use the absolute error err of the target position truth value D As a measurement indicator, the calculation method is shown in formula (3). truth Measured by a distance measuring instrument, the distance measurement value Dtest The experiment was conducted on the positioning error of pedestrians, electric vehicles and cars at a distance of 20m. D Statistics and analysis were performed, and the results are shown in Table 2
[0072] Table 2 Fusion positioning error test results
[0073]
[0074] Analyzing the data in the table reveals that at a distance of 20 meters, the fusion positioning error for electric vehicles and cars is generally within 0.2 meters, while the positioning error for pedestrians is larger. The superior fusion positioning results for electric vehicles and cars over pedestrians are due to the large size and high reflectivity of these two types of targets.
[0075] In order to see the error distribution more clearly, the fusion positioning error and its distribution of each frame at a distance of 20m for three types of targets are visualized here, as shown below: Figure 6 shown.
Claims
1. A target localization method based on the fusion of visual images and sparse point clouds, the method comprising: Step 1: Use a camera to obtain real-time 2D image data; Step 2: Use LiDAR to obtain real-time 3D scanning point cloud data; Step 3: The 2D image data obtained in step 1 and the 3D point cloud data obtained in step 2 are synchronized in time according to the timestamps, and spatial synchronization is performed using a joint calibration method; Step 4: Perform target detection on the 2D image data obtained in step 1 to obtain 2D image detection results, exclude small objects in the distance based on the pixel size of the detection frame; and use the SORT tracking method to perform multi-target tracking based on the detection results; Step 5: Divide the 3D scanning point cloud data obtained in step 2 into multiple sectors according to their spatial positions, with the scanning point as the center, and divide the point cloud data into sector rings within each sector; The sector ring segmentation method is: set the distance parameter and When each point satisfies formula (1), it is considered that the point is in the jth sector ring. Among them, (x i +y i ) is the projection coordinate of the i-th data point in the XY plane; Step 6: Convert the Cartesian coordinate system into a form similar to the polar coordinate system and perform straight line fitting on the dz plane; Where θ represents the angle between the laser line where the data point is located and the Z axis; Set the slope threshold of the laser line where the data point is located, the angle threshold between the laser line where the data point is located and the Z axis, and the straight line fitting error threshold of the laser line where the data point is located. Based on these thresholds, determine whether the data point is ground point cloud data and remove the ground point cloud data; Step 7: Segment the remaining point cloud data and cluster the segmented point cloud data to obtain the target's 3D information including position and size; Step 8: Associate the 2D detection results identified in step 4 with the 3D targets clustered in step 7. Map each target obtained in step 7 to the image using the visual projection matrix obtained in step 3. If the projection point falls within the 2D detection box, associate the 2D detection result with the 3D clustering result. Otherwise, the association fails.
2. The target positioning method based on post-fusion of visual images and sparse point clouds according to claim 1, characterized in that: In step 4, the multi-target tracking method SORT is improved. In addition to considering IOU during matching, target features are added, and whether they are the same target is determined by calculating the similarity of target features and IOU.
3. The target positioning method based on post-fusion of visual images and sparse point clouds according to claim 2, characterized in that: The specific method of step 7 is: Step 7.1: Create a KD-Tree, loop through each dimension of the 3D point cloud and select the median of the 3D point cloud in that dimension as the splitting hyperplane. Attach the 3D point cloud to the left of the median to the left subtree, and the 3D point cloud to the right of the median to the right subtree. Recursively process the subtrees until all point clouds are organized. Step 7.2: Access the data from the root node top to bottom, calculate the distance between the data and the leaf nodes, where the data point corresponding to the minimum distance Dis is recorded as Nearest, perform a backtracking operation to find the nearest neighbor point closer to the data, determine whether there is a point closer to the data in the unvisited branch, update the nearest neighbor point Nearest and the minimum distance Dis, specify the minimum and maximum search thresholds during the search process, abandon the segmentation of the point cloud type when the search result is less than the minimum threshold, and use the maximum search threshold for constraints when there are too many search results to complete the discrete point cloud instance segmentation; Step 7.3: Use the DBSCAN method to cluster the segmented point cloud data.
4. The target positioning method based on post-fusion of visual images and sparse point clouds according to claim 3, characterized in that: In step 8, the target point cloud obtained by clustering is projected onto the image through vision. If 60% of the laser points fall within the 2D image detection frame, the association is successful, otherwise the association fails.