An online multi-target tracking method based on visual point cloud fusion

Abstract

The present application belongs to the technical field of image processing, and particularly relates to an online multi-target tracking method based on visual point cloud fusion. The visual point cloud fusion online multi-target tracking method provided by the present application projects two-dimensional detection data to a camera coordinate system through coordinate system conversion, fuses image point cloud information, realizes the complementary advantages of a camera and a laser radar, and solves the problem of three-dimensional multi-target tracking difficulty in complex scenes such as high target density and low environmental brightness. The present application adopts a spherical model to represent a target for tracking, and compared with a traditional cuboid model, the present application reduces the calculation dimension, saves resources, improves speed, and is conducive to the deployment of the algorithm in an online tracking system. The present application is tested in a validation set of a public data set KITTI, and the tracking accuracy, precision and speed are obviously improved compared with other methods.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an online multi-target tracking method based on visual point cloud fusion. Background Technology

[0002] Multi-target tracking technology has made significant progress, aiming to solve the problem of effectively monitoring multiple targets in complex scenes. Traditional multi-target tracking systems mainly rely on a single sensor, such as a camera, LiDAR, or radar. These systems perform well in simple scenes, but face a series of challenges in complex and changing real-world environments. For example, cameras are susceptible to changes in lighting and occlusion, while LiDAR may perform poorly when dealing with non-planar or non-rigid targets. Furthermore, single-sensor systems often struggle to simultaneously meet the requirements of real-time performance, robustness, and accuracy.

[0003] To address these issues, researchers have proposed a multi-platform multi-target tracking scheme. By fusing information from multiple sensors, such as cameras and LiDAR, the system can capture target motion and features more comprehensively and accurately. This multi-platform approach aims to improve the system's robustness, making it more suitable for various complex scenarios. However, despite significant progress in multi-platform multi-target tracking technology, some shortcomings remain. First, the heterogeneity between data from different sensors makes data fusion difficult, requiring more sophisticated algorithms to handle features of different data types. Second, in complex environments, target trajectories vary greatly, and traditional linear filters may not be sufficient for accurate tracking of nonlinear motion. Furthermore, in large-scale, densely populated target scenarios, the increased demands on computational resources and real-time performance pose challenges to algorithm efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide an online multi-target tracking method based on visual point cloud fusion, which fuses visual point cloud information into a sphere, improving accuracy while reducing computational dimensionality, thereby achieving better real-time tracking results.

[0005] The technical solution adopted in this invention is as follows:

[0006] An online multi-target tracking method based on visual point cloud fusion includes the following steps:

[0007] S1. Acquire high-quality 3D multi-object detection result data as input data for 3D multi-object tracking. In frame t, the output of the 3D detection block is a set of detection... (n t (This refers to the number of tests), each test Represented by a tuple (f,t,x1,y1,x2,y2,s,h,w,l,x,y,z,θ), which includes the current frame number f, the object category t, the coordinates of the top left corner (x1,y1) and the bottom right corner (x2,y2) of the object's 2D image, the confidence score s, the object's 3D dimensions (h,w,l), the position of the object's center in 3D space (x,y,z), and the heading angle θ;

[0008] S2. Construct a sphere model by projecting the image detection data onto three-dimensional coordinates and fusing it with point cloud detection information to generate a sphere. The specific processing method is as follows:

[0009] a1. Select the t-th frame image F t A point on the k-th target Using formula

[0010] X camera =K -1 q

[0011] Mapping the image coordinates to the camera coordinate system yields the points. In the formula, X camera =(x,y,z) T Let q = (u, v, 1) be the coordinates of the point in the camera coordinate system. T Let K be the coordinates of the point in the image coordinate system, and K be the intrinsic parameter matrix of the camera, specifically...

[0012]

[0013] Among them, f x and f y c represents the camera's focal length in the horizontal and vertical directions of the image, respectively. x and c y These are the coordinates of the principal point, which define the optical center of the image;

[0014] a2. In the camera coordinate system, connect camera C and the projection point. And extend it to obtain the straight line l;

[0015] a3. Calculate the 3D detection box of the line l and the point cloud. The two intersections

[0016] a4, using line segments Draw a sphere with its midpoint as the center and its length as the diameter. Obtain a visual spherical model of the target's outer shell;

[0017] S3. Input the sphere model parameters from S2 into the Kalman filter for tracking. The tracking process is as follows:

[0018] a1. Represent the state of the target trajectory as a 9-dimensional vector T = (x, y, z, θ, r, s, v) x ,v y ,v z ), where the additional variable v x v y and v z This represents the target's velocity in three-dimensional space. In each frame, based on a uniform velocity model, the trajectory state associated with the previous frame is... (where m) t-1 The number of trajectories in frame t-1 will be propagated to frame t to obtain the predicted position:

[0019] x est =x+v x ,y est =y+v y ,z est =z+v z ,

[0020] Specifically, for each trajectory in Tt-1 (where i∈{1,2,…,m) t-1}), the predicted state in frame t is

[0021] a2. Calculate each pair of trajectories and detection The 3D intersection-over-union (IoU) or negative center distance between them is used to construct m t-1 ×n t The affinity matrix is ​​then determined. The data association is then transformed into a bipartite graph matching problem, solved using the Hungarian algorithm. Simultaneously, if the 3D IoU is less than a threshold IoU... min (or the center distance is greater than the threshold dist) max If the match fails, the matching is rejected. The data association output is as follows:

[0022]

[0023]

[0024]

[0025]

[0026] Among them, T match and D match w represents the matched trajectory and the detection, respectively. t This indicates the number of matches. Also, T unmatch and D unmatch This indicates unmatched trajectories and detections;

[0027] a3, Based on D match The corresponding detection in the process updates T. match The state of each trajectory. Following the Bayesian rule, each trajectory T... k Update state T in frame t t k =(x',y',z',θ',r',s',v') x ,v' y ,v' z ), where k∈{1,2,…,w t},yes and A weighted average of the states;

[0028] a4. For unmatched detections Until it in the next Bir min The frame is continuously matched, where p∈{1,2,…,n} t -w t Only then will a new trajectory be created. In ensuring It is a vanished trajectory. Previously, we continuously tracked each unmatched trajectory. Where q∈{1,2,…,m} t-1 -w t} and remove it from the associated trajectory set.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention provides an online multi-target tracking method based on visual point cloud fusion. By transforming the coordinate system, it projects 2D detection data onto the camera coordinate system and fuses image point cloud information, achieving complementary advantages between the camera and LiDAR. This solves the problem of difficult 3D multi-target tracking in complex scenarios such as high target density and low ambient brightness. Using a sphere model to represent targets for tracking reduces computational dimensionality compared to the traditional cuboid model, saving resources and increasing speed, which is beneficial for the algorithm's deployment in real-time tracking systems. The invention was tested on the publicly available KITTI dataset for 3D target tracking. The test results show that the accuracy of 3D multi-target tracking for vehicles and bicycles is improved by 3.89 and 3.78 percentage points respectively compared to the baseline method, demonstrating the performance advantage of this method. Attached Figure Description

[0031] Figure 1 A flowchart illustrating an online multi-target tracking method based on visual point cloud fusion, provided in an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the sphere model provided in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the multi-target tracking results of image point clouds in an embodiment of the present invention; Detailed Implementation

[0034] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0035] refer to Figure 1 This embodiment provides an online multi-target tracking method based on visual point cloud fusion, including:

[0036] Step 1: Use a pre-trained model of Point R-CNN on the KITTI dataset to obtain high-quality 3D multi-object detection results. In frame t, the output of the 3D detection block is a set of detections. (n t (This refers to the number of tests). Each test Represented by a tuple (f,t,x1,y1,x2,y2,s,h,w,l,x,y,z,θ), which includes the current frame number f, the object category t, the coordinates of the top left corner (x1,y1) and the bottom right corner (x2,y2) of the object's 2D image, the confidence score s, the object's 3D dimensions (h,w,l), the position of the object's center in 3D space (x,y,z), and the heading angle θ;

[0037] Step 2: Construct a sphere model, such as Figure 2 As shown, the center point of the 2D detection box is projected onto the 3D coordinate system as a reference point and fused with the point cloud detection information to generate a sphere. The specific processing method is as follows:

[0038] Step 2.1: Select the t-th frame image F t The center point of the 2D detection box on the k-th target. Using formula

[0039] X camera =K -1 q

[0040] Mapping the image coordinates to the camera coordinate system yields the points. In the formula, X camera =(x,y,z) T Let q = (u, v, 1) be the coordinates of the point in the camera coordinate system. T Let K be the coordinates of the point in the image coordinate system, and K be the intrinsic parameter matrix of the camera.

[0041]

[0042] Among them, f x and f y c represents the camera's focal length in the horizontal and vertical directions of the image, respectively. x and cy These are the coordinates of the principal point;

[0043] Step 2.2: In the camera coordinate system, connect camera C and the projection point. And extend it to obtain the straight line l;

[0044] Step 2.3: Calculate the 3D detection box of line l and point cloud. The two intersections

[0045] Step 2.4, using line segments A sphere is constructed with its midpoint as the center and its length as the diameter. From the visible spherical model of the target, the coordinates of the sphere's center can be calculated.

[0046]

[0047] Radius length is

[0048]

[0049] in, and Intersection points The coordinates;

[0050] Step 3: Input the sphere model parameters from Step 2 into the Kalman filter for tracking. The tracking process is as follows:

[0051] Step 3.1, State Prediction. To predict the target trajectory state from the previous frame to the current frame, this implementation uses a uniform velocity model. This model is independent of the camera's own motion and approximates the target's inter-frame displacement using a constant velocity model. That is, it does not explicitly estimate the camera's own motion but relies on a motion model to adapt to the camera's own motion and the motion of other targets. This implementation represents the target trajectory state as a 9-dimensional vector T = (x, y, z, θ, r, s, v). x ,v y ,v z ), where variable v x v y and v z This represents the target's velocity in three-dimensional space. In each frame, based on a uniform velocity model, the trajectory state associated with the previous frame is... (where m) t-1 The number of trajectories in frame t-1 will be propagated to frame t to obtain the predicted position:

[0052] x est =x+v x ,y est =y+v y ,z est =z+vz ,

[0053] Specifically, for each trajectory in Tt-1 (where i∈{1,2,…,m) t-1}), the predicted state in frame t is

[0054] Step 3.2, Data Association. To associate the predicted trajectory T... est With detection D t To perform matching, first calculate each pair of trajectories. and detection The 3D intersection-over-union (IoU) or negative center distance between them is used to construct m t-1 ×n t The affinity matrix is ​​then determined. The data association is then transformed into a bipartite graph matching problem, which is solved in polynomial time using the Hungarian algorithm. Simultaneously, if 3DIoU is less than the threshold IoU... min (or the center distance is greater than the threshold dist) max If the match fails, the matching is rejected. The data association output is as follows:

[0055]

[0056]

[0057]

[0058]

[0059] Among them, T match and D match w represents the matched trajectory and the detection, respectively. t This indicates the number of matches. Also, T unmatch and D unmatch This indicates unmatched trajectories and detections;

[0060] Step 3.3, State Update. To account for the uncertainty of state prediction, this implementation is based on D. match The corresponding detection in the process updates T. match The state of each trajectory in the data is used to obtain the final associated trajectory in frame t. Following Bayes' rule, each trajectory T k Update state T in frame t t k =(x',y',z',θ',r',s',v') x ,v' y ,v' z ), where k∈{1,2,…,w t},yes and The state-weighted average, with weights determined by the matched trajectory. and The uncertainty of the state is determined;

[0061] Step 3.4: Generating and Eliminating Memories. This implementation method will generate and eliminate all unmatched detections D. unmatch To avoid creating false positive trajectories, unmatched detections are treated as potential new objects entering the scene. Until it in the next Bir min The frame is continuously matched, where p∈{1,2,…,n} t -w t Only then will a new trajectory be created. Once the new trajectory It is created, and its state is initialized to its most recent detection. and v x v y and v z The speed is set to zero. Additionally, this implementation sets all unmatched trajectories T... unmatch Consider objects that may leave the scene, while ensuring It is a vanished trajectory. Previously, we continuously tracked each unmatched trajectory. Where q∈{1,2,…,m} t-1 -w t The track is then removed from the associated track set. Ideally, true positive examples with missing detection can be interpolated using the 3D multi-object tracking system of this embodiment without being deleted; only tracks that have actually left the scene are deleted.

[0062] The effectiveness of the proposed solution is described below using experiments on publicly available datasets.

[0063] As described in the above implementation scheme, this embodiment was evaluated on the KITTI dataset, which provides LiDAR point clouds and 3DBOX trajectories. Since the KITTI test set only supports 2D MOT evaluation and its ground truth is not released to users, the KITTI validation set (1, 6, 8, 10, 12, 13, 14, 15, 16, 18, 19) was used for 3D MOT evaluation. Regarding target categories, the KITTI convention was followed, displaying the results for each category (car, pedestrian, cyclist). The matching criteria followed the KITTI 3D object detection benchmark convention, using 3D IoU to determine successful matching. Since the KITTI ground truth is a cuboid, this embodiment combined the sphere tracking results with detection information to transform it into a cuboid for evaluation to facilitate the calculation of various metrics.

[0064] Regarding evaluation metrics, this implementation method uses standard CLEAR metrics for evaluation, such as MOTA (Multi-Target Tracking Accuracy), MOTP (Multi-Target Tracking Precision), IDS (Indicator Switching Count), FRA (Trajectory Breaking Count), and FPS (Frames Per Second), as well as sAMOTA (Scaled Average Multi-Target Tracking Accuracy), AMOTA (Average Multi-Target Tracking Accuracy), and AMOTP (Average Multi-Target Tracking Precision) proposed in the AB3DMOT paper. Table 1 shows the tracking performance of the car on the KITTI validation set.

[0065] Table 1 Comparison of car tracking performance on the KITTI validation set.

[0066]

[0067] It can be seen that when using different matching criteria (such as IoU) thres When the values ​​are 0.25, 0.5, and 0.7, the 3D multi-object tracking system of this embodiment outperforms other 3D multi-object tracking systems in all metrics, and still maintains the 0ID switching of the baseline algorithm AB3DMOT. It is also worth noting that the last column of Table 1 compares the inference time of all methods. The 3D multi-object tracking system of this embodiment (excluding the 3D detection part) runs at 765.3 FPS on the KITTI validation set without a GPU, achieving the fastest speed compared to other algorithms.

[0068] Tests show that, thanks to the visual shell sphere model, the 3D multi-target tracking method of this invention significantly improves various indicators by fusing image point cloud features and reducing computational dimensionality.

Claims

1. An online multi-target tracking method based on visual point cloud fusion, characterized in that, Includes the following steps: S1. Obtain the 3D multi-object detection results for 3D multi-object tracking. Defined in frame t, the 3D multi-object detection is expressed as... Where n t It refers to the number of tests, which is the number of tests per test. Let j ∈ 1, 2, ..., n be represented by a tuple (f, t, x1, y1, x2, y2, s, h, w, l, x, y, z, θ). t The tuple includes the current frame number f, the object category t, the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2) in the object's 2D image, the confidence score s, the object's 3D dimensions (h, w, l), the position of the object's center in 3D space (x, y, z), and the heading angle θ. S2. Construct a sphere model by projecting the image detection data onto three-dimensional coordinates and fusing it with point cloud detection information to generate a sphere. The specific processing method is as follows: a1. Select the t-th frame image F t A point on the k-th target Using formula X camera =K -1 q Mapping the image coordinates to the camera coordinate system yields the points. In the formula, X camera =(x,y,z) T Let q = (u, v, 1) be the coordinates of the point in the camera coordinate system. T Let K be the coordinates of the point in the image coordinate system, and K be the intrinsic parameter matrix of the camera. Among them, f x and f y c represents the camera's focal length in the horizontal and vertical directions of the image, respectively. x and c y These are the coordinates of the principal point; a2. In the camera coordinate system, connect camera C and the projection point. And extend it to obtain the straight line l; a3. Calculate the 3D detection box of the line l and the point cloud. The two intersections a4, using line segments The midpoint is the center of the sphere, and the line segment Make a sphere with length equal to diameter Obtain a visual spherical model of the target's outer shell; S3. Input the sphere model parameters from S2 into the Kalman filter for tracking. The tracking process is as follows: a1. Represent the state of the target trajectory as a 9-dimensional vector T = (x, y, z, θ, r, s, v) x ,v y ,v z ), where the additional variables vx and v y and v z This represents the target's velocity in three-dimensional space; in each frame, based on the uniform velocity model, the trajectory state associated with the previous frame is... Propagate to frame t, where m t-1 The predicted position is obtained by counting the number of trajectories in frame t-1. x est =x+v x ,y est =y+v y ,z est =z+v z , Specifically, for T t-1 Each trajectory in where i∈{1,2,…,m} t-1 The predicted state in frame t is: a2. Calculate each pair of trajectories and detection The 3D intersection-union ratio or negative center distance between them is used to construct m t-1 ×n t We then construct an affinity matrix and transform the data association into a bipartite graph matching problem, which is solved using the Hungarian algorithm. Simultaneously, if the 3D intersection-union ratio (IoU) is less than a threshold value... min Or the center distance is greater than the threshold dist max If the match fails, the matching is rejected; the data association output is as follows: Among them, T match and D match w represents the matched trajectory and the detection, respectively. t Indicates the number of matches, and T unmatch and D unmatch This indicates unmatched trajectories and detections; a3, Based on D match The corresponding detection in the process updates T. match The state of each trajectory; following the Bayesian rule, each trajectory T k Update state T in frame t t k =(x',y',z',θ',r',s',v') x ,v' y ,v' z ),yes and The weighted average of the states, where k∈{1,2,…,w} t }; a4. For unmatched detections Until it in the next Bir min A new trajectory is created only when the frames are continuously matched. Where p∈{1,2,…,n t -w t }; In ensuring It is a vanished trajectory. Previously, we continuously tracked each unmatched trajectory. And remove it from the set of associated trajectories, where q∈{1,2,…,m} t-1 -w t }

Images