Three-dimensional target detection and tracking method fusing multi-sensor information

Abstract

A three-dimensional target detection and tracking method fusing multi-sensor information belongs to the technical field of computer vision, and can make up for the defect of a single mode by collecting camera and radar image information and fusing camera images and laser radar point clouds. Multi-target tracking is carried out by using a multi-data coupling method of 2D + 3D data, a remote object is identified through a camera, when the object enters an induction range of the laser radar, the laser radar can carry out more accurate positioning, and position estimation of the target in a 2D image is corrected by using a more accurate 3D position provided by the laser radar. The method is used for solving the problems of inaccurate target identification, target shielding and the like under complex road conditions; a traditional single-mode three-dimensional target detection algorithm cannot effectively fuse the advantages of multi-sensor data and is difficult to accurately identify objects in a distance or under a complex road condition.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for improving a three-dimensional target detection and tracking framework. Background Art

[0002] With the rapid development of computer vision technology, autonomous driving technology has made significant progress. Traditional deep learning-based two-dimensional object detection can effectively classify and locate objects, but they can only provide information on the plane. In contrast, the latest 3D object detection technology can not only identify and locate objects, but also provide their spatial geometric information, which includes the actual size of the object and the precise distance from the autonomous vehicle.

[0003] Three-dimensional object detection technology significantly enhances the perception ability and decision-making accuracy of autonomous vehicles. By accurately measuring the distance between the vehicle and surrounding objects, the system can more reliably plan the driving path, predict possible collision risks and take corresponding avoidance measures, thereby significantly improving the safety and reliability of autonomous vehicles. In recent years, with the rapid development of lidar sensors and the great success of deep learning technologies represented by CNN and Transformer in the field of traffic scene perception, the current deep learning-based 3D object detection algorithms are divided into three types: camera RGB image-based, lidar point cloud-based, and multimodal 3D object detection based on RGB image-lidar point cloud fusion.

[0004] For multimodal 3D target detection and tracking technology that fuses RGB images and LiDAR point clouds, RGB images can provide rich texture and color information, and LiDAR point clouds can accurately capture the 3D structural information of the environment, but their sparsity and disorder will lead to a lack of semantics. In recent years, many detection frameworks with excellent performance have been developed in the field of 3D target detection. At present, there are three main ways to detect 3D targets: image-based 3D target detection algorithms, point cloud-based 3D target detection algorithms, and multimodal fusion 3D target detection algorithms. For fields such as autonomous driving, since point clouds contain accurate spatial information, LiDAR-based methods can achieve better performance than image-based methods. However, LiDAR points are usually sparse and do not have enough color and texture information. For image-based methods, they perform better in capturing semantic information, but lack depth signals. Therefore, multimodal 3D object detection is a promising direction that can make full use of the complementary information of images and point clouds. The detection framework based on multimodal fusion is more in line with the requirements of actual scenarios.

[0005] Three-dimensional target tracking plays a key role in the field of autonomous driving, such as environmental perception and obstacle detection, behavior prediction and path planning. Tracking of target objects helps ensure that the vehicle can safely and smoothly integrate into the traffic flow. However, target occlusion during the tracking process will also have many effects on the autonomous driving system, increasing the risk of collision and making path planning difficult. Therefore, a trajectory update strategy based on distance threshold is proposed, which can effectively reduce trajectory interruptions and effectively determine the occurrence of occlusion during trajectory sleep. This technology can be used as part of the autonomous driving system to effectively improve the safety of the autonomous driving process. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a three-dimensional target detection and tracking method that integrates multi-sensor information to solve the problems of inaccurate target recognition and target occlusion under complex road conditions; the traditional single-modality three-dimensional target detection algorithm cannot effectively integrate the advantages of multi-sensor data and it is difficult to accurately identify objects in the distance or under complex road conditions.

[0007] The three-dimensional target detection and tracking method integrating multi-sensor information includes the following steps:

[0008] Step 1: Obtain time-synchronized lidar point clouds and camera images in the same space through a lidar sensor and a camera;

[0009] Step 2: pre-process the laser radar point cloud data in step 1 and perform key point sampling on foreground points with more information;

[0010] Step 3: voxelize the original input lidar point cloud of step 1 and project it to obtain BEV features; perform cascade bidirectional fusion on the point cloud and camera image after key point sampling in step 2, and perform multi-head attention fusion with the obtained BEV features to obtain multi-scale and multi-source fusion features; use a three-dimensional target detector to obtain a three-dimensional target detection frame, and use a two-dimensional target detector to obtain a two-dimensional target detection frame;

[0011] Step 4: Associating the three-dimensional object detection frame and the two-dimensional object detection frame of the same object to obtain a group of instances;

[0012] Step 5: Update the target trajectory using the instance obtained in step 4 through a two-stage matching process. The 3D state of a target trajectory consists of its 3D target detection box and a position and velocity vector.

[0013] Step 6: Use 3D Kalman filter to perform motion modeling for motion prediction, update the state of the instance obtained in step 4, update the predicted state of the corresponding trajectory through matching detection, and obtain the matching trajectory of the final t frames;

[0014] Step 7: Compare the distance between the object and the lidar with the set distance threshold distthres, determine whether the object is out of the detection range or blocked, adjust the tracking termination threshold max_age, and re-detect the trajectory.

[0015] The key point sampling method is:

[0016] A lightweight point segmentation module is added to the SA layer to divide the output points into foreground and background. As input to the k-th SA layer l k dimensional point features, the foreground score of each point p i ∈[0,1], calculated as follows:

[0017]

[0018] Where Mk(·) represents the point segmentation module in the kth SA layer, and inputs the point-by-point feature f i Mapped to the foreground score p i ; σ(·) is the sigmoid function; any point within the true 3D bounding box is considered as a foreground point, and other points are considered as background points.

[0019] The multi-scale and multi-source fusion features use 3D sparse convolution to complete the feature extraction of four voxels of different scales, Fmix = [Fvoxel1×, Fvoxel2×, Fvoxel4×, Fvoxel8×, Fraw, FBEV], Fvoxel1×, Fvoxel2×, Fvoxel4×, Fvoxel8× are voxel features of different scales, Fraw is the point cloud feature, F BEV It is a BEV feature.

[0020] The instance is obtained by inputting the two-dimensional target detection result into the fusion module, projecting the three-dimensional target detection frame onto the plane, obtaining the intersection-over-union (IOU) of the projection and the two-dimensional detection frame, arranging the IOU values ​​of different combinations in descending order, and combining them into an instance when the following two conditions are met: t = {I 0 t ,…,I i t}, the instance contains a 3D bounding box and a 2D bounding box;

[0021] Condition 1: The IOU value exceeds the set threshold;

[0022] Condition 2: Neither the 2D detection results nor the 3D detection results in the combination match.

[0023] The two-stage matching includes a first-stage association and a second-stage association;

[0024] In the first phase, the 3D state and the 3D instance are associated. The 3D state is predicted by a constant velocity motion model constructed by Kalman filtering, and the reduced distance is used to determine whether the instance and the trajectory can be associated.

[0025]

[0026]

[0027] is a vector containing the 3D position and size, represents the direction of the bounding box around the vertical axis; the best matching instance and trajectory are put together to obtain 1m IT t ={(I i t , T j t ), ...}; no matching mark is an unmatched 3D instance 1u I t and unmatched 3D status 1u T t ; After this stage, the matched 3D instances 3d I t Will no longer participate in the second stage of matching;

[0028] In the second stage of association, except for the fusion instance both I t 2D instance outside 2d I t and unmatched 3D status 1u T t , 2D state 2d T t Associate; use 2D IOU to determine whether the instance and trajectory can be associated.

[0029] The 3D Kalman filter is used to perform motion modeling for motion prediction, and the motion prediction model is as follows:

[0030]

[0031] In the formula, the trajectory state of each target is represented by a ten-dimensional vector (x, y, z, θ, l, w, h, v x ,v y ,v z ), (x, y, z) represents the 3D coordinate center of the target, (l, w, h) represents the 3D size of the target, v x ,v y ,vz represents the speed of the target in three-dimensional space, θ represents the heading angle, t represents the current frame, and t+1 represents the predicted frame.

[0032] When the distance between the object and the laser radar exceeds the set distance threshold distthres, it exceeds the object detection range, and a smaller tracking termination threshold max_age1 is set. When the number of consecutive frame losses of the target exceeds the tracking termination threshold, the track is deleted, otherwise the track is restored;

[0033] When the distance between the object and the laser radar is less than or equal to the distance threshold distthres, the object is blocked by an obstacle, and a larger tracking termination threshold max_age2 is set to ensure that the target is not detected for a period of time and the track is still retained.

[0034] Through the above design scheme, the present invention can bring the following beneficial effects:

[0035] 1. Camera images provide rich visual information such as color and texture, which can effectively identify the shape, appearance characteristics and semantic information of objects. LiDAR point clouds provide high-precision spatial position and distance information, which can accurately measure the position, shape and relationship of objects with the environment. Fusion of camera images and LiDAR point clouds can make up for the defects of a single modality and effectively enhance the stability and accuracy of 3D target detection.

[0036] 2. Use the multiple data coupling method of 2D+3D data to track multiple targets. Use the camera to identify distant objects. When the object enters the sensing range of the lidar, the lidar can locate it more accurately. Use the more accurate 3D position provided by the lidar to correct the position estimate of the target in the 2D image. By fusing 2D and 3D data, the world coordinates and position of the target can be obtained more accurately.

[0037] 3. By setting the distance threshold in the trajectory update management process, the problem of target trajectory loss due to long-term occlusion during tracking can be effectively solved, thereby maintaining the tracking accuracy and stability, making the tracking system more adaptable to complex environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0039] Figure 1 This is a structural diagram of the multi-source data fusion framework for three-dimensional target detection in the present invention.

[0040] Figure 2 This is a structural diagram of the three-dimensional target tracking framework of multi-sensor fusion of the present invention.

[0041] Figure 3 This is a structural diagram of the trajectory update strategy based on the distance threshold of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention.

[0043] like Figure 1 As shown, the present invention proposes a multi-modal fusion three-dimensional target detection and tracking method, which can fuse the data provided by different sensors and simultaneously obtain information in two forms, namely, lidar point cloud and RGB image, so as to improve the detection and recognition capabilities of the target. Using 2D+3D data for multi-target tracking, the camera image can identify distant objects, and when the object enters the sensing range of the lidar, the lidar can perform more accurate positioning. The specific implementation steps are as follows:

[0044] Step 1: Obtain time-synchronized and spatially identical radar point clouds and camera images through a lidar sensor and camera, respectively.

[0045] Step 2: In order to focus on the foreground points with more information, the input LiDAR point cloud data is preprocessed, that is, key point sampling is performed. The specific steps are as follows:

[0046] A point binary segmentation module is added to the SA layer (set abstraction layer, which is responsible for abstracting and extracting local features from the point cloud) in Pointnet++ to identify foreground points from the input. It is a simple 2-layer multi-layer perceptron (MLP) and divides the input points into two categories, namely foreground and background. The specific representation is As the lk-dimensional point feature input to the k-th SA layer, the foreground score p of each point i ∈[0,1] is calculated as follows:

[0047]

[0048] where Mk(·) represents the point segmentation module in the kth SA layer, mapping the input point-wise features fi to the foreground score p i σ(·) is the sigmoid function. Any point within the ground-truth 3D bounding box is considered as a foreground point, and other points are considered as background points. The total segmentation loss is calculated using the cross entropy (CE) loss function:

[0049]

[0050] in and represents the predicted foreground score and ground truth segmentation label (1 for foreground and 0 for background) of the i-th point in the k-th SA layer. Nk and λk are the total number of input points and the segmentation loss weight of the k-th SA layer.

[0051] Step 3: voxelize the LiDAR point cloud and project it to obtain BEV features, and fuse the obtained BEV features, point cloud features, and image features. The specific process is as follows:

[0052] 1. First, the point cloud with semantic information sampled in step 2 is cascaded and bidirectionally fused with the camera image. The point-by-point correspondence between the 2D position and the 3D point cloud is established through the known projection matrix M. For a specific point P in the point set P, its corresponding position in the camera image is p'=Mp. Then bilinear interpolation is used to obtain the corresponding image features at continuous coordinates, which makes good use of the neighboring context information around the projection position. This process can be described as follows:

[0053]

[0054] 2. Connect the point cloud features and BEV features of all key point areas at different scales with multiple image features to obtain the mixed multi-source feature Fmix = [Fvoxel1×, Fvoxel2×, Fvoxel4×, Fvoxel8×, Fraw, FBEV]. Due to the different data dimensions of different feature information, a module composed of two layers of perception is used to adjust the feature dimensions of all features to the same. Use cross-multi-head attention for feature fusion to better model the global relationship between multi-source and multi-scale features. Specifically, the original point cloud features are used as input Q, and the mixed multi-source features Fmix are used as K and V.

[0055] 3. Use a 3D object detector to obtain a 3D object detection frame, and use a 2D object detector to obtain a 2D object detection frame.

[0056] Step 4: Associate the 2D and 3D detection results belonging to the same target to obtain a group of instances.

[0057] The specific steps are as follows:

[0058] First, the 2D and 3D detection results are input into the fusion module, the 3D detection box is projected onto the plane, and then the IOU of the projection and the 2D detection box is compared. The IOU values ​​of different combinations are arranged in descending order, and they are combined into an instance when the following two conditions are met: t = {I 0 t ,…,I i t}(contains a 3Dbounding box and a 2D boundingbox).

[0059] 1. The IOU value exceeds the set threshold

[0060] 2. The 2D detection results and 3D detection results in the combination do not match

[0061] The remaining unmatched detection results only contain 2D detection results or 3D detection results.

[0062] Step 5: Use the instance obtained in step 4 to update the target trajectory through a two-stage matching process. The 3D state of a target trajectory consists of its 3D detection box and a position and velocity vector.

[0063] In the first stage, the 3D state and the 3D instance are associated. The 3D state is predicted by a constant speed motion model constructed by Kalman filtering, and the reduced distance is used to determine whether the instance and the trajectory can be associated.

[0064]

[0065] The best matching instance-trajectory pairs are put together to obtain 1m IT t ={(I i t , T j t ), …}. No matching tag is 1u I t and 1u T t After this stage, the matched 3d I t Will no longer participate in the second stage of matching.

[0066] The second phase of association will 2d I t (excluding both I t , because it has a 3D bounding box, which has been matched in the first stage and is no longer involved in the second stage of matching) and 1u T t , 2d T t Use 2D IOU to determine whether instances and trajectories can be associated.

[0067] Step 6: Status Update

[0068] The present invention adopts a constant velocity model to approximate the displacement between target frames. The trajectory state of each target is represented by a ten-dimensional vector (x, y, z, θ, l, w, h, vx ,v y ,v z ), where (x, y, z) represents the 3D coordinate center of the target, (l, w, h) represents the 3D size of the target, and v x ,v y ,v z represents the speed of the target in three-dimensional space, θ represents the heading angle, t represents the current frame, and t+1 represents the predicted frame. The 3D Kalman filter is used to perform motion prediction. The motion prediction model is as follows:

[0069]

[0070] Use the Kalman filter to update the state with the instance matched in step 4. For the 2D trajectory state, directly use the latest detected 2D bounding box to cover it. For the 3D trajectory state, use the 3D background box plus a constant speed linear Kalman filter to update. If there is no 3D background box, use the result inferred by the Kalman filter of the previous frame to update.

[0071] Step 7: Track lifecycle management

[0072] 1. When the distance between the object and the lidar exceeds the set distance threshold (distthres), the system assumes that the object has not been detected and is beyond the object detection range, and then sets a smaller tracking termination threshold max_age1. When the number of consecutive frame drops of the target exceeds the tracking termination threshold, that is, time_since_update>max_age1, the track will be deleted, otherwise the track will be restored.

[0073] 2. When the distance between the object and the lidar is less than or equal to the distance threshold (distthres), the system assumes that the object is blocked by other obstacles rather than being undetected. In this case, a larger tracking termination threshold (max_age2) is applied to ensure that the track will not be deleted even if the target is not detected for a period of time.

[0074] Multiple tracking termination thresholds are set according to different distance thresholds (distthres). For targets at a longer distance, a shorter tracking termination threshold can be applied to save system resources and reduce unnecessary track persistence. For targets that may only be briefly occluded or temporarily leave the scene, the system prolongs the inactivity time of their tracks to avoid frequent track deletion and reconstruction.

Claims

1. A three-dimensional target detection and tracking method integrating multi-sensor information, characterized by: The following steps are included: Step 1: Obtain time-synchronized lidar point clouds and camera images in the same space through a lidar sensor and a camera; Step 2: pre-process the laser radar point cloud data in step 1 and perform key point sampling on foreground points with more information; Step 3: voxelize the original input lidar point cloud of step 1 and project it to obtain BEV features; perform cascade bidirectional fusion on the point cloud and camera image after key point sampling in step 2, and perform multi-head attention fusion with the obtained BEV features to obtain multi-scale and multi-source fusion features; use a three-dimensional target detector to obtain a three-dimensional target detection frame, and use a two-dimensional target detector to obtain a two-dimensional target detection frame; Step 4: Associating the three-dimensional object detection frame and the two-dimensional object detection frame of the same object to obtain a group of instances; Step 5: Update the target trajectory using the instance obtained in step 4 through a two-stage matching process. The 3D state of a target trajectory consists of its 3D target detection box and a position and velocity vector. Step 6: Use 3D Kalman filter to perform motion modeling for motion prediction, update the state of the instance obtained in step 4, update the predicted state of the corresponding trajectory through matching detection, and obtain the matching trajectory of the final t frames; Step 7: Compare the distance between the object and the lidar with the set distance threshold distthres, determine whether the object is out of the detection range or blocked, adjust the tracking termination threshold max_age, and re-detect the trajectory.

2. The three-dimensional target detection and tracking method based on multi-sensor information fusion according to claim 1 is characterized in that: The key point sampling method is: A lightweight point segmentation module is added to the SA layer to divide the output points into foreground and background. As input to the k-th SA layer l k dimensional point features, the foreground score of each point p i ∈[0,1], calculated as follows: Where Mk(·) represents the point segmentation module in the kth SA layer, and inputs the point-by-point feature f i Mapped to the foreground score p i ; σ(·) is the sigmoid function; any point within the true 3D bounding box is considered as a foreground point, and other points are considered as background points.

3. The three-dimensional target detection and tracking method based on multi-sensor information fusion according to claim 1 is characterized in that: The multi-scale and multi-source fusion features use 3D sparse convolution to complete the feature extraction of four voxels of different scales, Fmix = [Fvoxel1×, Fvoxel2×, Fvoxel4×, Fvoxel8×, Fraw, FBEV], Fvoxel1×, Fvoxel2×, Fvoxel4×, Fvoxel8× are voxel features of different scales, Fraw is the point cloud feature, F BEV It is a BEV feature.

4. The three-dimensional target detection and tracking method of fusing multi-sensor information according to claim 1 is characterized by: The instance is obtained by inputting the two-dimensional target detection result into the fusion module, projecting the three-dimensional target detection frame onto the plane, obtaining the intersection-over-union (IOU) of the projection and the two-dimensional detection frame, arranging the IOU values ​​of different combinations in descending order, and combining them into an instance when the following two conditions are met: t = {I 0 t ,…,I i t }, the instance contains a 3D bounding box and a 2D bounding box; Condition 1: The IOU value exceeds the set threshold; Condition 2: Neither the 2D detection results nor the 3D detection results in the combination match.

5. The three-dimensional target detection and tracking method of fusing multi-sensor information according to claim 1 is characterized in that: The two-stage matching includes a first-stage association and a second-stage association; In the first phase, the 3D state and the 3D instance are associated. The 3D state is predicted by a constant velocity motion model constructed by Kalman filtering, and the reduced distance is used to determine whether the instance and the trajectory can be associated. is a vector containing the 3D position and size, represents the direction of the bounding box around the vertical axis; the best matching instance and trajectory are put together to obtain 1m IT t ={(I i t , T j t ), ...}; no matching mark is an unmatched 3D instance 1u I t and unmatched 3D status 1u T t ; After this stage, the matched 3D instances 3d I t Will no longer participate in the second stage of matching; In the second stage of association, except for the fusion instance both I t External 2D instance 2d I t and unmatched 3D status 1u T t , 2D state 2d T t Associate; use 2D IOU to determine whether the instance and trajectory can be associated.

6. The three-dimensional target detection and tracking method of fusing multi-sensor information according to claim 1 is characterized by: The 3D Kalman filter is used to perform motion modeling for motion prediction, and the motion prediction model is as follows: In the formula, the trajectory state of each target is represented by a ten-dimensional vector (x, y, z, θ, l, w, h, v x ,v y ,v z ), (x, y, z) represents the 3D coordinate center of the target, (l, w, h) represents the 3D size of the target, v x ,v y ,v z represents the speed of the target in three-dimensional space, θ represents the heading angle, t represents the current frame, and t+1 represents the predicted frame.

7. The three-dimensional target detection and tracking method of fusing multi-sensor information according to claim 1 is characterized by: When the distance between the object and the laser radar exceeds the set distance threshold distthres, it exceeds the object detection range, and a smaller tracking termination threshold max_age1 is set. When the number of consecutive frame losses of the target exceeds the tracking termination threshold, the track is deleted, otherwise the track is restored; When the distance between the object and the laser radar is less than or equal to the distance threshold distthres, the object is blocked by an obstacle, and a larger tracking termination threshold max_age2 is set to ensure that the target is not detected for a period of time and the track is still retained.

Images