Three-dimensional multi-target tracking method based on cross-modal perception collaborative optimization

Abstract

The application discloses a three-dimensional multi-target tracking method based on cross-modal perception collaborative optimization, and belongs to the technical field of multi-target tracking.The application is used for solving the problem that a multi-target tracking method cannot effectively align 2D detection results and 3D detection results in space.The application comprises the following steps: obtaining a data matching pair of a 3D detection set and a 2D detection set, and optimizing mixed 3D data by using mixed 2D data;predicting a current state of a target by using an extended Kalman filter based on a historical track of the target, and obtaining a target state prediction value; performing track matching in three stages by using the target state prediction value, the optimized mixed 3D data, pure 3D detection data and pure 2D detection data in sequence, adjusting track confidence, and finally determining a current active track, so that multi-target tracking is realized.The application is used for multi-target tracking.

Description

Technical Field

[0001] This invention relates to a three-dimensional multi-target tracking method based on cross-modal perception and collaborative optimization, belonging to the field of multi-target tracking technology. Background Technology

[0002] 3D multi-object tracking is a highly valuable research branch in computer vision, aiming to depict the movement path of objects of interest in three-dimensional space. It has wide applications in smart warehousing, smart cities, aerospace, and augmented reality. For example, in smart warehousing, 3D multi-object tracking technology can accurately capture the dynamics of goods, AGVs, and other vehicles within the warehouse space, facilitating automated goods handling and inventory management, and ensuring efficient warehouse operations. In smart city construction, this technology can be used to monitor the three-dimensional trajectories of vehicles, pedestrians, and non-motorized vehicles on city roads, providing data support for traffic flow optimization and event early warning, thereby improving the intelligence level of urban governance.

[0003] In the field of 3D multi-target tracking, multimodal fusion of cameras and LiDAR has become a core means to improve tracking accuracy. Cameras assist in the localization of distant targets with rich appearance semantic information, while LiDAR provides accurate 3D spatial localization. The synergy between the two can effectively address tracking challenges in complex scenarios. However, current multimodal fusion frameworks still have significant limitations in the key stage of spatial alignment, which restricts further improvement in tracking performance.

[0004] At the spatial alignment level, existing methods generally face a contradiction between 3D detection accuracy and reliability: to ensure tracking recall, 3D object detectors typically use low-confidence thresholds to filter results, which inevitably introduces a large number of false positives, severely interfering with subsequent tracking processes. 2D detectors, due to their rich image appearance features, often provide more reliable detection results, but current technologies still utilize 2D information crudely—most methods only establish the correlation between 3D and 2D detections through simple weighted fusion or projection center point matching, failing to fully exploit their geometric consistency. For example, some methods directly minimize the Euclidean distance between the 3D and 2D detection center points, ignoring the overall spatial alignment of the bounding boxes, leading to inherent biases in optimization; other methods rely on manually designed weight parameters to balance modal differences, which not only increases parameter tuning complexity but also easily leads to decreased adaptability due to scene changes, preventing the 2D detection from fully correcting the 3D results.

[0005] In existing 3D multi-target tracking methods based on multimodal collaboration, the multimodal position optimizer achieves spatial alignment by constructing a "weighted Euclidean distance between the center point of the 3D detection box and the center point of the 2D detection box". The weights depend on manually set hyperparameters, which can easily lead to alignment errors. Furthermore, it only optimizes the center point distance and does not consider the overall spatial overlap of the bounding boxes.

[0006] In existing multimodal target recognition and tracking prediction methods based on LiDAR and binocular cameras, 3D detection involves filtering point clouds and then performing Euclidean clustering, followed by selecting the optimal cluster based on cluster size and distance weighting. If the point cloud is sparse (e.g., for distant targets), clustering may produce fragmented point cloud clusters, leading to inaccurate Box2 localization.

[0007] In summary, current multimodal 3D multi-target tracking methods still have considerable room for improvement in terms of spatial alignment refinement. Summary of the Invention

[0008] To address the problem that existing multi-target tracking methods cannot effectively align 2D and 3D detection results in space, this invention provides a three-dimensional multi-target tracking method based on cross-modal perception collaborative optimization.

[0009] The present invention provides a three-dimensional multi-target tracking method based on cross-modal sensing and cooperative optimization, comprising:

[0010] 2D similarity calculations are performed on the 3D and 2D detection sets of multiple targets to obtain data matching pairs, which are then used to construct a hybrid detection dataset. Other 3D detection data constitute a pure 3D detection dataset. Other 2D detection data constitute a pure 2D detection dataset. ;

[0011] For each data matching pair, the 3D data of the mixed detection is optimized using the 2D data of the mixed detection to obtain the optimized 3D data of the mixed detection;

[0012] The turning rate of each target and acceleration Treating it as a constant value, the target's current state is predicted using an extended Kalman filter based on the target's historical trajectory, and the predicted target state value is calculated.

[0013] The 3D generalized intersection-union ratio (GUU) is calculated based on the target state prediction and the optimized hybrid detection 3D data. The 2D GUU is calculated using the 2D projection of the target state prediction onto the camera plane and the 2D data of the hybrid detection, and a matching data cost matrix is ​​established. Based on the matching data cost matrix, the Hungarian algorithm is used to determine the first-stage matching trajectory, the first-stage unmatched trajectory, and the first-stage unmatched optimized hybrid detection 3D data.

[0014] For the unmatched trajectories in the first stage, the target state prediction value and the pure 3D detection dataset are compared. Calculate the pure 3D generalized intersection-union ratio and establish the pure 3D data cost matrix; based on the pure 3D data cost matrix, use the Hungarian algorithm to determine the target's second-stage matching trajectory, second-stage unmatched trajectory, and second-stage unmatched pure 3D data.

[0015] For the unmatched trajectories in the second stage, the predicted target state values ​​of the unmatched trajectories in the second stage are projected onto the image plane and compared with the pure 2D detection dataset. Calculate the pure 2D intersection-union ratio and establish the pure 2D data cost matrix; based on the pure 2D data cost matrix, use the Hungarian algorithm to determine the third-stage matching trajectory, the third-stage unmatched trajectory, and the third-stage unmatched pure 2D data for each target; the third-stage unmatched pure 2D data is deleted as invalid data.

[0016] The current trajectory is obtained by updating the historical trajectory based on the three-stage matching trajectory; each time the historical trajectory is updated, the confidence of the current trajectory is incremented by a preset step size.

[0017] The 3D data from the first stage of unmatched optimization and mixed detection, and the 3D data from the second stage of unmatched pure detection are initialized as new target temporary trajectories respectively; the stability of the new target temporary trajectory is judged, and if it is determined to be a real trajectory, it is used as the new target trajectory, and the confidence of the current trajectory is dynamically adjusted.

[0018] The stability of the unmatched trajectory in the third stage is judged. If it is determined to be a real trajectory, it is used as a new recovered trajectory; otherwise, it is discarded and terminated. At the same time, the confidence of the current trajectory is dynamically adjusted.

[0019] The updated current trajectory, new target trajectory, and newly recovered trajectory are judged based on the current trajectory confidence level to determine the current active trajectory and achieve multi-target tracking.

[0020] According to the three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization of the present invention, the method for obtaining data matching pairs is as follows:

[0021] 3D detection data from the 3D detection set are projected onto the image plane to obtain 2D projection boxes. The 2D similarity between the 2D projection boxes and the 2D detection data in the 2D detection set is calculated based on the intersection-over-union ratio (IoU).

[0022] ,

[0023] In the formula Represents the similarity matrix. Indicates intersection, union, and ratio. This represents the projection from three-dimensional space onto the image plane. This represents the 3D detection set. Represents a 2D detection set;

[0024] Based on the similarity matrix The Hungarian algorithm is used to obtain data matching pairs.

[0025] The method for obtaining optimized hybrid detection 3D data according to the cross-modal sensing cooperative optimization method of the present invention includes:

[0026] 3D detection set Data detected at time t Represented as ,in The coordinates of the center point of the 3D bounding box. In order, the target width, target length, and target height are: The heading angle of the 3D bounding box; the 2D detection set Data detected at time t Represented as ,in The coordinates of the top left corner of the target are: The coordinates of the bottom right corner of the target;

[0027] Establish objective function :

[0028] ;

[0029] The objective function is optimized using the Newton-Raphson optimization method. In Optimization and updates will be performed.

[0030] ;

[0031] ,

[0032] ,

[0033] ,

[0034] In the formula for The result of the k-th iteration is updated. and As an intermediate variable, The function is the Newton-optimized algorithm function;

[0035] When the difference between the results of two adjacent iterations is less than When the iteration process terminates, the optimized hybrid detection 3D data is obtained.

[0036] According to the three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization of the present invention, the method for obtaining the target state prediction value includes:

[0037] The target trajectory state is represented as a 10-dimensional vector. ,in This represents the coordinates of the geometric center of the target trajectory in 3D space. The target width, target length, and target height are shown in the target trajectory status, in that order. The target velocity in the target trajectory state. The target heading angle in the target trajectory state;

[0038] Predicting the target's current state using an extended Kalman filter:

[0039] ,

[0040] In the formula Given the target state at time t based on time t-1 The predicted current state of the target Let be the predicted value of the target state covariance matrix at time t. Let be the target state covariance matrix at time t-1. Given the Jacobian matrix, the state transition function is used. right Obtained by taking the partial derivative; This is process noise;

[0041] In the state transition of the motion model, the CTRA motion model is adopted, and the variables are assumed. If kept constant, the predicted value of the target state is:

[0042] ,

[0043] ,

[0044] In the formula Let x be the predicted x-axis coordinate of the geometric center of the target state at time t. Let x be the x-axis coordinate of the geometric center of the target trajectory at time t-1 in 3D space. The time interval between adjacent data frames. For integration variables, The target velocity in the target trajectory state The angle between the x-axis and the x-axis ; Let y be the predicted value of the geometric center of the target state at time t. Let y be the geometric center position of the target trajectory at time t-1 in 3D space.

[0045] , , The time interval between adjacent moments;

[0046] .

[0047] According to the three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization of the present invention, the method for establishing the matching data cost matrix is ​​as follows:

[0048] ,

[0049] ,

[0050] In the formula For 3D generalized intersection and union ratio, For the current state of the target The target state prediction value, To optimize the 3D data of the post-hybrid detection, For volume sets, For convex hull operations in 3D space. For 2D generalized intersection-union ratio, For the current state of the target Target state prediction values ​​2D projection frame data, For 2D data of mixed detection, For the set of areas, For convex hull operations in 2D space;

[0051] The matching data cost matrix is ​​represented as follows :

[0052] ,

[0053] In the formula For 3D balance parameters, These are 2D balance parameters.

[0054] According to the three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization of the present invention, a threshold for the cost matrix is ​​set in the Hungarian algorithm. In the case of a threshold with a cost matrix The Hungarian algorithm is used to determine the first-stage matching trajectory of the target on the matching data cost matrix. Unmatched trajectories in the first phase 3D data from mixed detection after the first stage of mismatch optimization .

[0055] According to the three-dimensional multi-target tracking method based on cross-modal sensing and cooperative optimization of the present invention, the method for establishing the pure 3D data cost matrix is ​​as follows:

[0056] ,

[0057] In the formula For pure 3D detection datasets Data in the middle;

[0058] The pure 3D data cost matrix is ​​represented as :

[0059] ;

[0060] Set matching threshold With matching threshold The Hungarian algorithm is used to determine the second-stage matching trajectory of the target using the pure 3D data cost matrix. Second phase unmatched trajectory Second phase unmatched pure 3D data .

[0061] According to the three-dimensional multi-target tracking method based on cross-modal sensing and cooperative optimization of the present invention, the method for establishing the pure 2D data cost matrix is ​​as follows:

[0062] ,

[0063] In the formula For pure 2D detection datasets Data in the middle;

[0064] The cost matrix of pure 2D data is represented as :

[0065] ;

[0066] Set matching threshold With matching threshold The Hungarian algorithm is used to determine the third-stage matching trajectory for each target based on the pure 2D data cost matrix. The third stage of unmatched trajectories and the third stage of unmatched pure 2D data .

[0067] According to the three-dimensional multi-target tracking method based on cross-modal sensing and collaborative optimization of the present invention, the method for determining the stability of the temporary trajectory of a new target is as follows:

[0068] If the new target temporary trajectory is successfully matched in the next adjacent iteration, then the new target temporary trajectory will be used as the new target trajectory, and the confidence of the current trajectory will be incremented by a preset step size.

[0069] According to the three-dimensional multi-target tracking method based on cross-modal sensing and collaborative optimization of the present invention, the method for determining the stability of unmatched trajectories in the third stage is as follows:

[0070] For unmatched trajectories in the third stage, if no trajectory matching result is obtained within 3 frames of detection data after the maximum duration, the corresponding trajectory segment is discarded and terminated; the confidence of the current trajectory is decremented by a preset step size; otherwise, it is treated as a new recovered trajectory, and the confidence of the current trajectory is incremented by a preset step size.

[0071] The beneficial effects of this invention are as follows: The method proposed in this invention proposes a preprocessing approach based on cross-modal sensing alignment, which can fully unleash the collaborative potential of multimodal data and improve the accuracy and efficiency of 3D multi-target tracking. It fully leverages the enhancing effect of 2D detection on 3D detection, effectively reducing errors, improving accuracy, and increasing computational efficiency.

[0072] This invention's method associates 3D and 2D detection boxes, optimizes the 3D detection box position using Newton's method, and iteratively adjusts the coordinates of the 3D box's center point to spatially align the 3D and 2D detection results. It employs a three-stage association logic, matching historical trajectories in the order of mixed detection → pure 3D detection → pure 2D detection, gradually reducing unmatched trajectories and detection results, thus improving association robustness. This invention directly maximizes the overlap area between the projected 3D bounding box and the 2D detection box. "Alignment is achieved without weight adjustment, naturally solving the unit difference problem, and the error is reduced by optimizing the overall overlap of the bounding boxes."

[0073] The method of this invention employs a multi-stage processing optimization process, progressively refining the processing to improve target localization and tracking accuracy. In motion model prediction, a Kalman filter approach is used. The overall boundary of the 3D bounding box is optimized using Newton's method (rather than just cluster center points), directly minimizing the difference between the 3D and 2D bounding boxes. Even with sparse point clouds, the semantic information of 2D images can constrain the position of 3D bounding boxes, thus improving positioning accuracy. Attached Figure Description

[0074] Figure 1 This is an overall flowchart of the three-dimensional multi-target tracking method based on cross-modal perception and collaborative optimization described in this invention;

[0075] Figure 2 This is a flowchart of the data processing for 3D and 2D detection sets.

[0076] Figure 3 This is a data processing block diagram of the three-dimensional multi-target tracking method based on cross-modal perception and collaborative optimization described in this invention. Detailed Implementation

[0077] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] Specific Implementation Method 1: Combination Figures 1 to 3 As shown, this invention provides a three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization, comprising:

[0079] 2D similarity calculations are performed on the 3D and 2D detection sets of multiple targets to obtain data matching pairs, which are then used to construct a hybrid detection dataset. Other 3D detection data constitute a pure 3D detection dataset. Other 2D detection data constitute a pure 2D detection dataset. ;

[0080] For each data matching pair, the 3D data of the mixed detection is optimized using the 2D data of the mixed detection to obtain the optimized 3D data of the mixed detection;

[0081] The turning rate of each target and acceleration Treating it as a constant value, the target's current state is predicted using an extended Kalman filter based on the target's historical trajectory, and the predicted target state value is calculated.

[0082] The 3D generalized intersection-union ratio (GUU) is calculated based on the target state prediction and the optimized hybrid detection 3D data. The 2D GUU is calculated using the 2D projection of the target state prediction onto the camera plane and the 2D data of the hybrid detection, and a matching data cost matrix is ​​established. Based on the matching data cost matrix, the Hungarian algorithm is used to determine the first-stage matching trajectory, the first-stage unmatched trajectory, and the first-stage unmatched optimized hybrid detection 3D data.

[0083] For the unmatched trajectories in the first stage, the target state prediction value and the pure 3D detection dataset are compared. Calculate the pure 3D generalized intersection-union ratio and establish the pure 3D data cost matrix; based on the pure 3D data cost matrix, use the Hungarian algorithm to determine the target's second-stage matching trajectory, second-stage unmatched trajectory, and second-stage unmatched pure 3D data.

[0084] For the unmatched trajectories in the second stage, the predicted target state values ​​of the unmatched trajectories in the second stage are projected onto the image plane and compared with the pure 2D detection dataset. Calculate the pure 2D intersection-union ratio and establish the pure 2D data cost matrix; based on the pure 2D data cost matrix, use the Hungarian algorithm to determine the third-stage matching trajectory, the third-stage unmatched trajectory, and the third-stage unmatched pure 2D data for each target; the third-stage unmatched pure 2D data is deleted as invalid data.

[0085] The current trajectory is obtained by updating the historical trajectory based on the three-stage matching trajectory; each time the historical trajectory is updated, the confidence of the current trajectory is incremented by a preset step size.

[0086] The 3D data from the first stage of unmatched optimization and mixed detection, and the 3D data from the second stage of unmatched pure detection are initialized as new target temporary trajectories respectively; the stability of the new target temporary trajectory is judged, and if it is determined to be a real trajectory, it is used as the new target trajectory, and the confidence of the current trajectory is dynamically adjusted.

[0087] The stability of the unmatched trajectory in the third stage is judged. If it is determined to be a real trajectory, it is used as a new recovered trajectory; otherwise, it is discarded and terminated. At the same time, the confidence of the current trajectory is dynamically adjusted.

[0088] The updated current trajectory, new target trajectory, and newly recovered trajectory are judged based on the current trajectory confidence level to determine the current active trajectory and achieve multi-target tracking.

[0089] In this embodiment, a data preprocessing module is constructed to obtain optimized 3D data from hybrid detection; a motion estimation module is constructed to calculate the predicted target state; a data association module is constructed to match the trajectory with the data; and a lifecycle management module is constructed to process the output of the data association module to obtain the target tracking trajectory. The data preprocessing module achieves efficient cross-modal spatiotemporal information integration. The method follows the detection-to-track (TBD) paradigm, seamlessly integrating mainstream detectors without additional training, and can optimally integrate LiDAR and camera data, such as... Figure 1 As shown. First, the projected 3D detection boxes and 2D detection boxes are matched using the Hungarian algorithm to obtain a hybrid detection dataset. Pure 3D detection dataset and pure 2D detection dataset For mixed detection datasets For each matching pair in the dataset, the 3D center coordinates are iteratively optimized using Newton's method to minimize the objective function until the detection box overlap rate meets the iterative update constraint; for unmatched pure 3D detection datasets... It can implement confidence score-based filtering and non-maximum suppression (NMS) to remove low-quality detections in pure 2D detection datasets. Retain directly for new target discovery, such as Figure 2As shown; then, an extended Kalman filter (EKF) is used to predict the trajectory state, and the position is calculated using the integral equation of motion; then, a three-stage data association (fusion) is performed. →pure →Projection Finally, the lifecycle management module dynamically manages the trajectory, such as... Figure 3 As shown.

[0090] Data preprocessing module based on cross-modal sensing alignment:

[0091] In each frame, existing detection algorithms are used to acquire 3D and 2D detection results. The 3D detection boxes are then projected onto the image plane and compared with the 2D detection results. The matching yields a hybrid detection dataset. Pure 3D detection dataset and pure 2D detection dataset Hybrid detection dataset Optimization is performed using geometrically perceptual alignment to produce a corrected output; pure 3D detection dataset. Fraction-based filtering and nonmaximum suppression (NMS) are applied to discard unreliable results; pure 2D detection dataset. It is directly passed to the downstream module without further processing.

[0092] To ensure high recall in tracking, 3D object detectors typically use low-confidence thresholds to filter false positives. This inevitably leads to a large number of false positives, severely impacting subsequent tracking tasks. Generally, 2D detectors are more reliable than 3D detectors because they can utilize richer appearance information. Therefore, this embodiment uses 2D detection results to correct 3D detection results. A method based on... This method employs a cross-modal alignment optimization approach (CLOUD) to fully leverage cross-modal synergy and improve overall performance, achieving enhanced 3D detection through 2D detection. Unlike traditional methods that rely on weighted fusion, this approach directly maximizes the alignment between the projected 3D bounding box and the 2D detection results, correcting 3D detection results using 2D detection results. No empirical weight adjustment is required. Specifically, cross-modal detection association is first performed. Given a set of 3D and 2D detection results, the correspondence between modalities is established by calculating the similarity between the 3D bounding boxes and the 2D bounding boxes projected onto the image plane.

[0093] Furthermore, the method for obtaining data matching pairs is as follows:

[0094] 3D detection data from the 3D detection set are projected onto the image plane to obtain 2D projection boxes. The 2D similarity between the 2D projection boxes and the 2D detection data in the 2D detection set is calculated based on the intersection-over-union ratio (IoU).

[0095] ,

[0096] In the formula Represents the similarity matrix. Indicates intersection, union, and ratio. This represents the projection from three-dimensional space onto the image plane. This represents the 3D detection set. Represents a 2D detection set;

[0097] Based on the similarity matrix The Hungarian algorithm is used to obtain data matching pairs.

[0098] A link between 2D and 3D detections was established by obtaining data matching pairs. Following the association step, 3D detections were optimized using 2D detections. For each matched 3D and 2D detection, the 3D center was optimized to maximize the overlap between the projected 3D bounding box and the 2D detection bounding box. Maximize, and then define the objective function for optimization.

[0099] Methods for obtaining optimized hybrid detection 3D data include:

[0100] 3D detection set Data detected at time t Represented as ,in The coordinates of the center point of the 3D bounding box. In order, the target width, target length, and target height are: The heading angle of the 3D bounding box; the 2D detection set Data detected at time t Represented as ,in The coordinates of the top left corner of the target are: The coordinates of the bottom right corner of the target;

[0101] Establish objective function :

[0102] ;

[0103] Optimization is a nonlinear problem, and Newton's method is chosen for numerical optimization because it has the advantage of fast convergence.

[0104] The objective function is optimized using the Newton-Raphson optimization method. In Optimization and updates will be performed.

[0105] ;

[0106] ,

[0107] ,

[0108] ,

[0109] In the formula for The result of the k-th iteration is updated. and As an intermediate variable, The function is the Newton-optimized algorithm function;

[0110] When the difference between the results of two adjacent iterations is less than At this point, the iterative process terminates, yielding optimized 3D data from the hybrid detections. This produces the optimal 3D coordinate center. This center is then used to replace the original 3D detections for correction. The overall optimization stops within several to dozens of iterations, resulting in lower time costs compared to data association and motion estimation. Finally, the corrected 3D detections are input into the motion module for joint motion modeling.

[0111] Motion estimation module:

[0112] The turning rate of each target and acceleration It is considered a constant value, and the target's heading angle and motion pattern are closely coupled, that is, the target's heading angle in the target trajectory state. ,speed and acceleration The directions are in a straight line, applicable to both car targets and pedestrians.

[0113] Furthermore, methods for obtaining the predicted value of the target state include:

[0114] The target trajectory state is represented as a 10-dimensional vector. ,in This represents the coordinates of the geometric center of the target trajectory in 3D space. The target width, target length, and target height are shown in the target trajectory status, in that order. The target velocity in the target trajectory state. The target heading angle in the target trajectory state;

[0115] Due to the nonlinear characteristics of the motion model, the extended Kalman filter is used to predict the current state of the target:

[0116] ,

[0117] In the formula Given the target state at time t based on time t-1 The predicted current state of the target Let be the predicted value of the target state covariance matrix at time t. Let be the target state covariance matrix at time t-1. Given the Jacobian matrix, the state transition function is used. right Obtained by taking the partial derivative; This is process noise, and its value is manually set. It is a state transition function established by the motion model, reflecting the changes of all state variables of the trajectory between two consecutive frames.

[0118] In the state transition of the motion model, the CTRA motion model is adopted, and the variables are assumed. If kept constant, the predicted value of the target state is:

[0119] ,

[0120] ,

[0121] In the formula Let x be the predicted x-axis coordinate of the geometric center of the target state at time t. Let x be the x-axis coordinate of the geometric center of the target trajectory at time t-1 in 3D space. The time interval between adjacent data frames. For integration variables, The target velocity in the target trajectory state The angle between the x-axis and the x-axis ; Let y be the predicted value of the geometric center of the target state at time t. Let y be the y-axis coordinate of the geometric center position of the target trajectory at time t-1 in 3D space.

[0122] To better illustrate the state transition process of variables over time in each motion model, a time interval is introduced in this embodiment. , , The integral variable during the integration process and the lower limit of integration The distance between them. The speed during the target position transfer. It is a linear change that follows time, that is , Angle with x-axis With heading angle Maintain consistency, heading angle transfer function .

[0123] Build a data association module:

[0124] To minimize matching failures, a three-stage data association is employed. The first stage of association addresses the issue that in complex scenes, due to the rapid movement of the target, the cost matrix calculated using only 3D positional information lacks discriminative power against nearby interfering objects, easily leading to incorrect matches. Therefore, a cost matrix is ​​created using both 2D and 3D detection results. For the trajectory in frame t-1... The expected state of frame t Then, the 3D generalized intersection-union ratio and the 2D generalized intersection-union ratio are used to correlate the optimized 3D data of the mixed detection and the 2D data of the mixed detection.

[0125] The method for establishing the cost matrix of matching data is as follows:

[0126] ,

[0127] ,

[0128] In the formula For 3D generalized intersection and union ratio, For the current state of the target The target state prediction value, To optimize the 3D data of the post-hybrid detection, For volume sets, For convex hull operations in 3D space. For 2D generalized intersection-union ratio, For the current state of the target Target state prediction values ​​2D projection frame data, For 2D data of mixed detection, For the set of areas, For convex hull operations in 2D space;

[0129] The matching data cost matrix is ​​represented as follows :

[0130] ,

[0131] In the formula For 3D balance parameters, These are 2D balance parameters.

[0132] In the Hungarian algorithm, a threshold is set for the cost matrix. In the case of a threshold with a cost matrix The Hungarian algorithm is used to determine the first-stage matching trajectory of the target on the matching data cost matrix. Unmatched trajectories in the first phase 3D data from mixed detection after the first stage of mismatch optimization .

[0133] Second-stage association: For unmatched trajectories from the first stage Further in and pure 3D detection dataset The correlations are established between them. Since the pure 3D detection dataset lacks corresponding 2D detection results, the cost matrix is ​​only obtained through... create.

[0134] The method for constructing a pure 3D data cost matrix is ​​as follows:

[0135] ,

[0136] In the formula For pure 3D detection datasets Data in the middle;

[0137] The pure 3D data cost matrix is ​​represented as :

[0138] ;

[0139] Set matching threshold With matching threshold The Hungarian algorithm is used to determine the second-stage matching trajectory of the target using the pure 3D data cost matrix. Second phase unmatched trajectory Second phase unmatched pure 3D data .

[0140] The third stage of association: To reduce premature trajectory termination and provide guidance for 3D prediction optimization, the pure 2D detection dataset is further integrated into the 2D space. Mismatched trajectory in Phase 2 To establish the association, firstly, a projection matrix needs to be used to... The 3D prediction is projected onto the image plane, and then... Construct the cost matrix;

[0141] The method for constructing the cost matrix of pure 2D data is as follows:

[0142] ,

[0143] In the formula For pure 2D detection datasets Data in the middle;

[0144] The cost matrix of pure 2D data is represented as :

[0145] ;

[0146] Set matching threshold With matching threshold The Hungarian algorithm is used to determine the third-stage matching trajectory for each target based on the pure 2D data cost matrix. The third stage of unmatched trajectories and the third stage of unmatched pure 2D data .

[0147] Lifecycle Management Module:

[0148] The method for determining the stability of a new target's temporary trajectory is as follows:

[0149] If the new target temporary trajectory is successfully matched in the next adjacent iteration, then the new target temporary trajectory will be used as the new target trajectory, and the confidence of the current trajectory will be incremented by a preset step size.

[0150] The method for determining the stability of unmatched trajectories in the third stage is as follows:

[0151] For unmatched trajectories in the third stage, if no trajectory matching result is obtained within 3 frames of detection data after the maximum duration, the corresponding trajectory segment is discarded and terminated; the confidence of the current trajectory is decremented by a preset step size; otherwise, it is treated as a new recovered trajectory, and the confidence of the current trajectory is incremented by a preset step size.

[0152] The lifecycle management module is responsible for four key functions, including trajectory update, trajectory initialization, trajectory termination, and output results.

[0153] The trajectory update process utilizes detection results and a standard update procedure based on extended Kalman filtering to update the state and covariance matrix of the corresponding trajectory, while simultaneously maintaining a continuous hit counter for the trajectory. Each successful match increments the counter value, quantifying the trajectory's sustained tracking stability. Furthermore, the trajectory confidence parameter is dynamically adjusted based on the hit frequency. A successful match increases the confidence level by a preset step size, enhancing the trajectory's reliable representation; conversely, a failed match decreases the trajectory confidence level.

[0154] Trajectory initialization: A counting-based method is used to initialize the unmatched detection results into a new temporary trajectory. If the j-th Hit consecutively in the next frame; temporary trajectory The temporary trajectory will become an active trajectory and be merged into the still active trajectory. The initial confidence level of the temporary trajectory is set to the basic threshold. After it becomes an active trajectory, it will be dynamically calibrated based on subsequent tracking performance.

[0155] Track termination: A counting-based method is used to discard tracks. .if Tracks that have not been updated in the past 3 frames are discarded. Tracks that are not discarded are still considered active and their confidence scores are penalized by setting a decay coefficient between 0 and 1. The more frames that have not been updated, the stronger the confidence penalty.

[0156] Output Results: After obtaining all active trajectories for the current frame, the updated trajectories (estimated motion states), newly initialized trajectories, and some penalized active trajectories are output to the results file. Furthermore, the output data must include unique trajectory identifiers, timestamps, and confidence parameters to provide complete metadata support for subsequent trajectory analysis.

[0157] The lifecycle management module achieves refined control over the target trajectory from generation to termination through multi-dimensional state quantification and dynamic adjustment mechanisms. While ensuring tracking continuity, it effectively suppresses false positive annotations and improves the engineering practicality of the multi-target tracking system.

[0158] In summary, the method of the present invention has the following advantages:

[0159] Significantly improved geometric alignment accuracy: Compared to traditional methods that minimize the Euclidean distance between the center points of the bounding boxes, the method of this invention directly minimizes the intersection-union ratio between the projected 3D bounding box and the 2D detection box. This achieves more accurate cross-modal geometric alignment. Existing methods using Euclidean distance metrics only reflect the deviation in the center point position, while... As a comprehensive metric based on region overlap, this invention simultaneously considers the combined effects of positional offset, size difference, and orientation deviation on projection consistency. In 3D target tracking tasks, reducing the center point projection error does not guarantee the spatial consistency between the 3D bounding box and the 2D detection box, especially when the target is rotated or partially occluded. The method employed in this invention... The optimization strategy, by maximizing the proportion of the overlapping area, essentially constrains the global consistency between the projected bounding box and the ground truth detection box, thereby reducing geometric alignment error.

[0160] Optimization of computational efficiency and cross-modal compatibility: As a normalization metric, its value range is limited to The proposed method, without requiring additional normalization, is applicable to both 3D space (meters) and 2D image planes (pixels). In contrast, existing methods using Euclidean distance have inherent limitations in cross-modal optimization: the distance deviation (meters) in 3D space and the distance deviation (pixels) in the 2D pixel plane exhibit a dimensional mismatch, necessitating the introduction of heuristic weight coefficients for manual balancing. Such weight adjustments not only increase the complexity of algorithm tuning but also introduce subjective bias. This invention addresses this issue by... The inherent normalization property of Newton's method completely avoids the problem of cross-modal dimensional unification, eliminating the need for any empirical parameter adjustments during the optimization process. Meanwhile, Newton's method... The fast convergence characteristics on the curved surface (average number of iterations ≤ 15 times) significantly improve the real-time performance of the system.

[0161] The method of this invention fully utilizes the advantage of rich semantic information in 2D detectors to solve the problem of false positives in 3D detection.

[0162] The lifecycle management module outputs a full-process control mechanism with metadata and is processed by NMS, ensuring the continuity of target tracking and the purity of results.

[0163] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A three-dimensional multi-target tracking method based on cross-modal sensing and collaborative optimization, characterized in that, include: 2D similarity calculations are performed on the 3D and 2D detection sets of multiple targets to obtain data matching pairs, which are then used to construct a hybrid detection dataset. Other 3D detection data constitute a pure 3D detection dataset. Other 2D detection data constitute a pure 2D detection dataset. ; For each data matching pair, the 3D data of the mixed detection is optimized using the 2D data of the mixed detection to obtain the optimized 3D data of the mixed detection; The turning rate of each target and acceleration Treating it as a constant value, the target's current state is predicted using an extended Kalman filter based on the target's historical trajectory, and the predicted target state value is calculated. The 3D generalized intersection-union ratio (GUU) is calculated based on the target state prediction and the optimized hybrid detection 3D data. The 2D GUU is calculated using the 2D projection of the target state prediction onto the camera plane and the 2D data of the hybrid detection, and a matching data cost matrix is ​​established. Based on the matching data cost matrix, the Hungarian algorithm is used to determine the first-stage matching trajectory, the first-stage unmatched trajectory, and the first-stage unmatched optimized hybrid detection 3D data. For the unmatched trajectories in the first stage, the target state prediction value and the pure 3D detection dataset are compared. Calculate the generalized intersection-union ratio (GUOR) of pure 3D data and establish the pure 3D data cost matrix; based on the pure 3D data cost matrix, use the Hungarian algorithm to determine the target's second-stage matching trajectory, second-stage unmatched trajectory, and second-stage unmatched pure 3D data. For the unmatched trajectories in the second stage, the predicted target state values ​​of the unmatched trajectories in the second stage are projected onto the image plane and compared with the pure 2D detection dataset. Calculate the pure 2D intersection-union ratio and establish the pure 2D data cost matrix; based on the pure 2D data cost matrix, use the Hungarian algorithm to determine the third-stage matching trajectory, the third-stage unmatched trajectory, and the third-stage unmatched pure 2D data for each target; In the third stage, unmatched pure 2D data is deleted as invalid data. The current trajectory is obtained by updating the historical trajectory based on the three-stage matching trajectory; each time the historical trajectory is updated, the confidence of the current trajectory is incremented by a preset step size. The 3D data from the first stage of unmatched optimization and mixed detection, and the 3D data from the second stage of unmatched pure detection are initialized as new target temporary trajectories respectively; the stability of the new target temporary trajectory is judged, and if it is determined to be a real trajectory, it is used as the new target trajectory, and the confidence of the current trajectory is dynamically adjusted. The stability of the unmatched trajectory in the third stage is judged. If it is determined to be a real trajectory, it is used as a new recovered trajectory. Otherwise, the trajectory is terminated as a discarded trajectory, and the confidence level of the current trajectory is dynamically adjusted. The updated current trajectory, new target trajectory, and newly recovered trajectory are judged based on the current trajectory confidence level to determine the current active trajectory and achieve multi-target tracking.

2. The three-dimensional multi-target tracking method based on cross-modal sensing and collaborative optimization according to claim 1, characterized in that, The method for obtaining data matching pairs is as follows: 2D projection boxes are obtained by projecting 3D detection data from the 3D detection set onto the image plane. The 2D similarity between the 2D projection boxes and the 2D detection data in the 2D detection set is calculated based on the intersection-over-union ratio (IoU). ， In the formula Represents the similarity matrix. Indicates intersection, union, and ratio. This represents the projection from three-dimensional space onto the image plane. This represents the 3D detection set. Represents a 2D detection set; Based on the similarity matrix The Hungarian algorithm is used to obtain data matching pairs.

3. The three-dimensional multi-target tracking method based on cross-modal sensing and collaborative optimization according to claim 1, characterized in that, Methods for obtaining optimized hybrid detection 3D data include: 3D detection set Data detected at time t Represented as ,in The coordinates of the center point of the 3D bounding box. In order, the target width, target length, and target height are: The heading angle of the 3D bounding box; the 2D detection set Data detected at time t Represented as ,in The coordinates of the top left corner of the target are: The coordinates of the bottom right corner of the target; Establish objective function : ； The objective function is optimized using the Newton-Raphson optimization method. In Optimization and updates will be performed. ； ， ， ， In the formula for The result of the k-th iteration is updated. and As an intermediate variable, The function is the Newton-optimized algorithm function; When the difference between the results of two adjacent iterations is less than When the iteration process terminates, the optimized hybrid detection 3D data is obtained.

4. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 3, characterized in that, Methods for obtaining the target state prediction include: The target trajectory state is represented as a 10-dimensional vector. ,in This represents the coordinates of the geometric center of the target trajectory in 3D space. The target width, target length, and target height are shown in the target trajectory status, in that order. The target velocity in the target trajectory state. The target heading angle in the target trajectory state; Predicting the target's current state using an extended Kalman filter: ， In the formula Given the target state at time t based on time t-1 The predicted target current state, Let be the predicted value of the target state covariance matrix at time t. Let be the target state covariance matrix at time t-1. Given the Jacobian matrix, the state transition function is used. right Obtained by taking the partial derivative; This is process noise; In the state transition of the motion model, the CTRA motion model is adopted, and the variables are assumed. If kept constant, the predicted value of the target state is: ， ， In the formula Let x be the predicted x-axis coordinate of the geometric center of the target state at time t. Let x be the x-axis coordinate of the geometric center of the target trajectory at time t-1 in 3D space. The time interval between adjacent data frames. For integration variables, The target velocity in the target trajectory state The angle between the x-axis and the x-axis ; Let y be the predicted value of the geometric center of the target state at time t. Let y be the geometric center position of the target trajectory at time t-1 in 3D space. , , The time interval between adjacent moments; 。 5. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 4, characterized in that, The method for establishing the cost matrix of matching data is as follows: ， ， In the formula For 3D generalized intersection and union ratio, For the current state of the target The target state prediction value, To optimize the 3D data of the post-hybrid detection, For volume sets, For convex hull operations in 3D space. For 2D generalized intersection-union ratio, For the current state of the target Target state prediction values ​​2D projection frame data, For 2D data of mixed detection, For the set of areas, For convex hull operations in 2D space; The matching data cost matrix is ​​represented as follows : ， In the formula For 3D balance parameters, These are 2D balance parameters.

6. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 5, characterized in that, In the Hungarian algorithm, a threshold is set for the cost matrix. In the case of a threshold with a cost matrix The Hungarian algorithm is used to determine the first-stage matching trajectory of the target on the matching data cost matrix. Unmatched trajectories in the first phase 3D data from mixed detection after the first stage of mismatch optimization .

7. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 6, characterized in that, The method for constructing a pure 3D data cost matrix is ​​as follows: ， In the formula For pure 3D detection datasets Data in the middle; The pure 3D data cost matrix is ​​represented as : ； Set matching threshold With matching threshold The Hungarian algorithm is used to determine the second-stage matching trajectory of the target using the pure 3D data cost matrix. Second phase unmatched trajectory Second phase unmatched pure 3D data .

8. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 7, characterized in that, The method for constructing the cost matrix of pure 2D data is as follows: ， In the formula For pure 2D detection datasets Data in the middle; The cost matrix of pure 2D data is represented as : ； Set matching threshold With matching threshold The Hungarian algorithm is used to determine the third-stage matching trajectory for each target based on the pure 2D data cost matrix. The third stage of unmatched trajectories and the third stage of unmatched pure 2D data .

9. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 8, characterized in that, The method for determining the stability of a new target's temporary trajectory is as follows: If the new target temporary trajectory is successfully matched in the next adjacent iteration, then the new target temporary trajectory will be used as the new target trajectory, and the confidence of the current trajectory will be incremented by a preset step size.

10. The three-dimensional multi-target tracking method based on cross-modal sensing cooperative optimization according to claim 8, characterized in that, The method for determining the stability of unmatched trajectories in the third stage is as follows: For unmatched trajectories in the third stage, if no trajectory matching result is obtained within 3 frames of detection data after the maximum duration, the corresponding trajectory segment is discarded and terminated; the confidence of the current trajectory is decremented once by a preset step size. Otherwise, it is treated as a new recovered trajectory, and the confidence of the current trajectory is incremented once by a preset step size.

Images