A three-dimensional multi-target online tracking method based on feature fusion and distance fusion

Abstract

This invention discloses a three-dimensional multi-target online tracking method based on feature fusion and distance fusion. The steps are as follows: First, target detection data of each target is acquired using a target detection algorithm; then, the fused features and positional features of the target are extracted; subsequently, the positional distance and feature distance between the trajectory and the detection are calculated; then, the feature distance weight is calculated and the final distance matrix is ​​calculated based on this weight; finally, greedy matching is used to perform data association based on the final distance matrix between the trajectory and the detection to obtain the multi-target tracking trajectory of the current frame. This invention extracts and fuses multimodal features of the target, enabling features that fully express the properties of the target in different environments to be extracted; this invention improves the accuracy of multi-target tracking by fusing information from multiple modalities at different levels; and the fusion process does not require manual intervention, avoiding a large number of manual experiments to search for suitable thresholds, thus saving a lot of time while improving the accuracy of multi-target tracking.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and computer vision, specifically to a three-dimensional multi-target online tracking method based on feature fusion and distance fusion, which is particularly suitable for target tracking in complex outdoor scenarios. Background Technology

[0002] With the continuous development of computer technology, target tracking technology has gradually become a key development area in intelligent video analysis and multimedia applications (such as intelligent surveillance, motion analysis, and autonomous driving). It enables the detection and tracking of multiple targets in complex scenes, real-time analysis and prediction of target trajectories, and provides richer data information for practical applications.

[0003] Thanks to the rapid iteration of computer hardware and deep learning technology, image and point cloud-based object detection technologies have become increasingly mature. Accurate object detection results provide stronger data support for multi-object tracking technologies. In real-world scenarios, the accuracy of multi-object tracking algorithms is often affected by changes in the external environment. Due to complex interactions and occlusions between targets, as well as changes in external environments such as lighting and rain, single-modal features of a target are insufficient to fully express its state and properties. Furthermore, the high-dimensional semantic features and low-dimensional physical features of a target are not suitable for direct fusion on the feature plane; if more modal features are utilized, the issue of fusion hierarchy needs to be addressed.

[0004] To address the aforementioned issues, features from different modalities of the target are extracted and their weights are adjusted and fused together using a squeezing excitation module, allowing features from each modality to play a role and improving the final tracking accuracy. Target motion is modeled to obtain motion information and thus the distance between targets, further integrating more modal information at the decision-making level. A feature distance weight estimation network is learned through a data-driven approach to accurately determine the weights of feature distances between targets, reducing the time spent on manual experiments and improving efficiency. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of existing multi-target tracking technologies, this invention provides a three-dimensional multi-target online tracking method based on feature fusion and distance fusion. This method has high recognition and tracking accuracy and is robust to complex target interactions and changes in the external environment.

[0006] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0007] A three-dimensional multi-target online tracking method based on feature fusion and distance fusion includes the following steps:

[0008] Step 1: Apply an object detector to detect vehicles in the current image sequence and point cloud sequence to obtain detection data;

[0009] Step 2: Extract the two-dimensional features, three-dimensional features, and position information of the target in the current frame;

[0010] Step 3: Fuse the two-dimensional and three-dimensional features of the target in the current frame to obtain the fused features of the target; use Kalman filtering to predict the position of the survival trajectory in the current frame;

[0011] Step 4: Calculate the Euclidean distance between the fused feature distance and the location of all targets in the current frame and the historical trajectory;

[0012] Step 5: Fuse the feature distance and Euclidean distance between targets;

[0013] Step 6: Perform data association based on the distance between targets.

[0014] Furthermore, step 1 specifically includes the following steps:

[0015] The DSA-PV-RCNN object detector is selected to perform object detection on the current input image sequence and point cloud sequence. The object detector is trained based on the public dataset KITTI, and the detection category is vehicles. M bounding boxes of the current frame are obtained: Where α represents the angle of the target relative to the sensor, u1 and v1 represent the x and y coordinates of the upper left corner of the 2D bounding box in the image, u2 and v2 represent the x and y coordinates of the lower right corner of the 2D bounding box in the image, h, w and l represent the height, width and length of the 3D bounding box, x, y and z represent the coordinates of the center point of the 3D bounding box in the point cloud, and r represents the rotation angle of the 3D bounding box.

[0016] Furthermore, step 2 specifically includes the following steps:

[0017] Step 2.1: Based on the detection data The corresponding location in the image is cropped and scaled to a fixed size of 224×224; then this image is input into a 2D feature extraction network to extract the 2D features of the target. Performing this operation on all M targets yields the two-dimensional features of each target, which can be represented as follows:

[0018] Step 2.2: Based on the detection data (h) i ,w i ,l i ,x i ,y i ,z i ,r iThe points at the corresponding locations in the point cloud are extracted and input into a 3D feature extraction network to extract the 3D features of the target. Performing this operation on all M targets yields the three-dimensional features of each target, which can be represented as follows:

[0019] Step 2.3: Based on the (x) in the detection data i ,y i ,z i This allows us to obtain the three-dimensional position information of the target.

[0020] Furthermore, step 3 specifically includes the following steps:

[0021] Step 3.1: Use the squeeze excitation module N1 to adjust the channel weights of the target two-dimensional feature to obtain new target two-dimensional features.

[0022]

[0023] Step 3.2: Use the extrusion excitation module N2 to adjust the channel weights of the target's three-dimensional features to obtain new target three-dimensional features.

[0024]

[0025] Step 3.3: Concatenate the new two-dimensional features and three-dimensional features of the target to obtain the fused features of the target.

[0026]

[0027] Indicates feature splicing;

[0028] Step 3.4: Based on the detection data within the historical trajectory set, establish the state equation for the tracked target, consisting of 6 variables: (x,y,z) represents the three-dimensional position information of the target. This represents the difference between the target's 3D position information and the 3D position information of the previous frame; then, Kalman filtering is applied to predict the trajectory's 3D position information in the current frame.

[0029] Furthermore, step 4 specifically includes the following steps:

[0030] Step 4.1: Calculate the feature distance between the i-th target in the current frame and the j-th trajectory in the historical trajectory using the feature distance estimation network N3.

[0031]

[0032] Let represent the fused features of the i-th target in the t-th frame, i.e., the current frame. This shows the fused features of the j-th historical trajectory among all historical trajectories up to frame t-1. The feature distance matrix can be obtained by calculating the feature distances between the historical trajectories and all targets in the current frame. M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame;

[0033] Step 4.2: Based on the predicted location obtained in Step 3.4 and the detection location extracted in Step 2.3, the Euclidean distance can be calculated.

[0034]

[0035] (x i ,y i ,z i () represents the 3D position information of the i-th target in the current frame. This represents the predicted position of the j-th historical trajectory in the current frame; calculating the Euclidean distance between the historical trajectories and all targets in the current frame yields the Euclidean distance matrix. M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame;

[0036] Furthermore, step 5 specifically includes the following steps:

[0037] Step 5.1: Calculate the feature distance weights A using the feature distance weight estimation network N4:

[0038]

[0039] These are the target fusion features calculated in step 3.3;

[0040] Step 5.2: Based on this weight, perform weighted fusion of feature distances and Euclidean distances between targets:

[0041]

[0042] Here, * indicates that corresponding elements are multiplied.

[0043] Furthermore, step 6 specifically includes the following steps:

[0044] Based on the distance matrix D between the current frame target and the historical trajectory obtained in step 5 finalStarting from the first row of the matrix, select the column with the lowest distance. If the distance is less than 2, it is considered that the target represented by this row and this column corresponds to the target and is assigned the same identity (ID). The column is also set to 100 to prevent secondary matching. If the distance is greater than 2, it is considered that the target represented by this row has no historical target that matches it and a new ID is assigned to it.

[0045] Beneficial effects: The three-dimensional multi-target online tracking method based on feature fusion and distance fusion provided by this invention has the following advantages.

[0046] 1) Based on the accurate detection results of DSA-PV-RCNN, in addition to the positional information of the bounding box itself, the extracted high-dimensional semantic features can more fully express the target features.

[0047] 2) The squeeze excitation module based on the attention mechanism adjusts the channel weights of the two-dimensional and three-dimensional features of the target, so that a more important part of the original features is emphasized and plays a more important role in the subsequent calculation process.

[0048] 3) The fused features, which incorporate multimodal features, can better express the state and properties of the target, thus enhancing the robustness of the algorithm in complex scenarios.

[0049] 4) At the decision-making level, the Euclidean distance obtained from target motion modeling is integrated, which utilizes more modal information of the target and further enhances the robustness of the algorithm in complex scenarios.

[0050] 5) Using a data-driven approach to learn the feature distance weight estimation network reduces the time spent manually searching for suitable parameters and improves the final tracking performance. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of the three-dimensional multi-target online tracking method based on feature fusion and distance fusion provided by the present invention;

[0052] Figure 2 This is a network structure diagram of the two-dimensional feature extraction network provided by the present invention;

[0053] Figure 3 This is a network structure diagram of the three-dimensional feature extraction network provided by the present invention;

[0054] Figure 4 This is a network structure diagram of the extrusion excitation module used in this invention;

[0055] Figure 5 This is an algorithm flowchart of the fusion module provided by the present invention;

[0056] Figure 6 This is a network structure diagram of the feature distance estimation network provided by the present invention.

[0057] Figure 7 This is a diagram of the feature distance weight estimation network structure provided by the present invention.

[0058] Figure 8 This is a flowchart of the data association algorithm used in this invention.

[0059] Figure 9 This is the dataset tracking result of the three-dimensional multi-target online tracking method based on feature fusion and distance fusion provided by this invention.

[0060] Figure 10 This is a visualization of the tracking results of the three-dimensional multi-target online tracking method based on feature fusion and distance fusion provided by the present invention. The upper figure is the visualization result of the 115th frame of video sequence 0 in the KITTI dataset, and the lower figure is the visualization result of the 100th frame of video sequence 2. Detailed Implementation

[0061] The invention will now be further described with reference to the accompanying drawings.

[0062] As shown in the figure, a three-dimensional multi-target online tracking method based on feature fusion and distance fusion includes the following steps:

[0063] Step 1: Apply an object detector to detect vehicles in the current image sequence and point cloud sequence to obtain detection data;

[0064] Step 2: Extract the two-dimensional features, three-dimensional features, and position information of the target in the current frame;

[0065] Step 3: Fuse the two-dimensional and three-dimensional features of the target in the current frame to obtain the fused features of the target; use Kalman filtering to predict the position of the survival trajectory in the current frame;

[0066] Step 4: Calculate the Euclidean distance between the fused feature distance and the location of all targets in the current frame and the historical trajectory;

[0067] Step 5: Fuse the feature distance and Euclidean distance between targets;

[0068] Step 6: Perform data association based on the distance between targets.

[0069] In this embodiment, step 1 specifically includes the following steps:

[0070] The DSA-PV-RCNN object detector is selected to perform object detection on the current input image sequence and point cloud sequence. The object detector is trained based on the public dataset KITTI, and the detection category is vehicles. M bounding boxes of the current frame are obtained: Where α represents the angle of the target relative to the sensor, u1 and v1 represent the x and y coordinates of the upper left corner of the 2D bounding box in the image, u2 and v2 represent the x and y coordinates of the lower right corner of the 2D bounding box in the image, h, w and l represent the height, width and length of the 3D bounding box, x, y and z represent the coordinates of the center point of the 3D bounding box in the point cloud, and r represents the rotation angle of the 3D bounding box.

[0071] In this embodiment, step 2 specifically includes the following steps:

[0072] Step 2.1: Based on the detection data The corresponding location in the image is cropped and scaled to a fixed size of 224×224; then this image is input into a 2D feature extraction network to extract the 2D features of the target. Performing this operation on all M targets yields the two-dimensional features of each target, which can be represented as follows:

[0073] Step 2.2: Based on the detection data (h) i ,w i ,l i ,x i ,y i ,z i ,r i The points at the corresponding locations in the point cloud are extracted and input into a 3D feature extraction network to extract the 3D features of the target. Performing this operation on all M targets yields the three-dimensional features of each target, which can be represented as follows:

[0074] Step 2.3: Based on the (x) in the detection data i ,y i ,z i This allows us to obtain the three-dimensional position information of the target.

[0075] In this embodiment, step 3 specifically includes the following steps:

[0076] Step 3.1: Use the squeeze excitation module N1 to adjust the channel weights of the target two-dimensional feature to obtain new target two-dimensional features.

[0077]

[0078] Step 3.2: Use the extrusion excitation module N2 to adjust the channel weights of the target's three-dimensional features to obtain new target three-dimensional features.

[0079]

[0080] Step 3.3: Concatenate the new two-dimensional features and three-dimensional features of the target to obtain the fused features of the target.

[0081]

[0082] Indicates feature splicing;

[0083] Step 3.4: Based on the detection data within the historical trajectory set, establish the state equation for the tracked target, consisting of 6 variables: (x,y,z) represents the three-dimensional position information of the target. This represents the difference between the target's 3D position information and the 3D position information of the previous frame; then, Kalman filtering is applied to predict the trajectory's 3D position information in the current frame.

[0084] In this embodiment, step 4 specifically includes the following steps:

[0085] Step 4.1: Calculate the feature distance between the i-th target in the current frame and the j-th trajectory in the historical trajectory using the feature distance estimation network N3.

[0086]

[0087] Let represent the fused features of the i-th target in the t-th frame, i.e., the current frame. This shows the fused features of the j-th historical trajectory among all historical trajectories up to frame t-1. The feature distance matrix can be obtained by calculating the feature distances between the historical trajectories and all targets in the current frame. M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame;

[0088] Step 4.2: Based on the predicted location obtained in Step 3.4 and the detection location extracted in Step 2.3, the Euclidean distance can be calculated.

[0089]

[0090] (x i ,y i ,z i () represents the 3D position information of the i-th target in the current frame. This represents the predicted position of the j-th historical trajectory in the current frame; calculating the Euclidean distance between the historical trajectories and all targets in the current frame yields the Euclidean distance matrix. M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame;

[0091] In this embodiment, step 5 specifically includes the following steps:

[0092] Step 5.1: Calculate the feature distance weights A using the feature distance weight estimation network N4:

[0093]

[0094] These are the target fusion features calculated in step 3.3;

[0095] Step 5.2: Based on this weight, perform weighted fusion of feature distances and Euclidean distances between targets:

[0096]

[0097] Here, * indicates that corresponding elements are multiplied.

[0098] In this embodiment, step 6 specifically includes the following steps:

[0099] Based on the distance matrix D between the current frame target and the historical trajectory obtained in step 5 final Starting from the first row of the matrix, select the column with the lowest distance. If the distance is less than 2, it is considered that the target represented by this row and this column corresponds to the target and is assigned the same identity (ID). The column is also set to 100 to prevent secondary matching. If the distance is greater than 2, it is considered that the target represented by this row has no historical target that matches it and a new ID is assigned to it.

[0100] Example

[0101] This invention presents a 3D multi-target online tracking method based on feature fusion and distance fusion. It utilizes a DSA-PV-RCNN target detector to acquire detection data from image sequences and point cloud sequences, with vehicles selected as the target type for multi-target tracking. The training dataset for both the detector and tracker is the KITTI dataset, which includes 8008 images and 24070 vehicles. The backbone network of the algorithm's 2D feature extraction network is VGGNet, and the backbone network of the 3D feature extraction network is PointNet. The training losses for the distance estimation network and the feature distance weight estimation network are contrast loss and page folding loss.

[0102] Experiments: The algorithm was tested on the KITTI dataset to verify the effectiveness of the proposed method for multi-target tracking.

[0103] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A three-dimensional multi-target online tracking method based on feature fusion and distance fusion, characterized by comprising the following steps: Step 1: applying a target detector to detect vehicles in a current image sequence and a point cloud sequence to obtain detection data; Step 2: extracting two-dimensional features, three-dimensional features and position information of the target in the current frame; Step 3: fusing the two-dimensional features and the three-dimensional features of the target in the current frame to obtain fused features of the target; and using Kalman filtering to predict the position of the survival trajectory in the current frame; Step 4: calculating the fused feature distance and the Euclidean distance between the position of all targets between the current frame and the historical trajectory; Step 5: fusing the feature distance and the Euclidean distance between the targets; Step 6: data association according to the distance between the targets. The specific method of step 1 is as follows: The target detector DSA-PV-RCNN performs target detection on the current input image sequence and point cloud sequence, and the training of the target detector is based on a public dataset KITTI, and the detection category is vehicle; M detection data bounding box data of the current frame are obtained: Wherein, a represents the angle of the target relative to the sensor, u1 and v1 represent the horizontal and vertical coordinates of the upper left corner of the two-dimensional bounding box in the image, u2 and v2 represent the horizontal and vertical coordinates of the lower right corner of the two-dimensional bounding box in the image, h, w and l represent the height, width and length of the three-dimensional bounding box, x, y and z represent the coordinates of the center point of the three-dimensional bounding box in the point cloud, and r represents the rotation angle of the three-dimensional bounding box. The specific method of step 2 is as follows: Step 2.1: According to the detection data in The picture of the corresponding position in the image is cropped and scaled to a fixed size of 224x224; then the picture is input into a two-dimensional feature extraction network to extract the two-dimensional features of the target This operation is performed on all M targets to obtain the two-dimensional features of each target, denoted as Step 2.2: According to (h i ,w i ,l i ,x i ,y i ,z i ,r i ) in the detection data, the point corresponding to the position in the point cloud is taken out and input into the three-dimensional feature extraction network to extract the three-dimensional feature of the target This operation is performed on all M targets to obtain the three-dimensional feature of each target, denoted as Step 2.3: Obtain the three-dimensional position information of the target according to (x i ,y i ,z i ) in the detection data; The specific method of step 3 is as follows: Step 3.1: Adjust the target two-dimensional feature channel weight using the extrusion incentive module N1 to obtain a new target two-dimensional feature Step 3.2: Adjust the target three-dimensional feature channel weight using the extrusion incentive module N2 to obtain a new target three-dimensional feature Step 3.3: Splicing the target new two-dimensional feature with the three-dimensional feature to obtain the fusion feature of the target representing feature stitching; Step 3.4: Establish the state equation of the tracked target based on the detection data within the historical trajectory set, consisting of 6 variables: (x, y, z) represents the three-dimensional position information of the target, represents the difference between the three-dimensional position information of the target and the three-dimensional position information of the previous frame; then apply Kalman filtering to predict the three-dimensional position information of the trajectory in the current frame 2.The method of claim 1, wherein, The specific method of step 4 is as follows: Step 4.1: Calculate the feature distance between the i-th target in the current frame and the j-th trajectory in the historical trajectories by the network N3 denotes the fusion feature of the i-th target in the t-th frame, i.e. the current frame, denotes the fusion feature of the j-th historical trajectory in all historical trajectories until the t-1-th frame. Calculate the feature distance between the historical trajectory and all targets in the current frame to obtain a feature distance matrix M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame. Step 4.2: Calculate the Euclidean distance based on the predicted position from Step 3.4 and the detected position extracted in Step 2.3 (x i ,y i ,z i ) represents the three-dimensional position information of the i-th target in the current frame, represents the predicted position of the j-th historical trajectory in the current frame; the Euclidean distance between the historical trajectory and all targets in the current frame is calculated to obtain the Euclidean distance matrix M represents the number of targets in the current frame, and N represents the number of targets in the adjacent previous frame. 3.The method of claim 2, wherein, The specific method of step 5 is as follows: Step 5.1: calculating the feature distance weight A by using a feature distance weight estimation network N4: is the target fusion feature calculated in step 3.3; Step 5.2: weighting and fusing the feature distance and the Euclidean distance between the targets according to the weight: Wherein, * represents multiplication of corresponding elements. 4.The method of claim 3, wherein, The specific method of step 6 is as follows: The distance matrix D between the current frame target and the historical trajectory obtained according to step 5 final From the first row of the matrix, the column with the lowest distance is selected. If the distance is less than 2, it is considered that the target represented by this row and this column is corresponding, the same identity ID is assigned, and the column is set to 100 to prevent secondary matching. If the distance is higher than 2, it is considered that the target represented by this row has no historical target to match, and a new ID is assigned.

Images