A multi-target tracking method for an automatic driving scene and a car

Abstract

The application discloses to the technical field of automatic driving perception, specifically is a kind of multi-target tracking method and car for automatic driving scene;Through the acquisition vehicle-mounted sensor continuous frame three-dimensional point cloud and two-dimensional image, generate three-dimensional detection frame by fusing 3D detector, realize target trajectory tracking relying on UMRTrack model.State estimation adopts online noise adaptive Kalman filtering, combined with the geometric, semantic information of detection frame dynamically constructs observation noise covariance, adjusts filter gain in real time, and relieves state drift.Data association is through detection quality score adaptive correction mahalanobis distance threshold, fuses geometry and motion cost to construct hybrid cost matrix, completes first round of matching by bird's-eye view through hungarian algorithm, and remaining target is based on point cloud density cross-view projection secondary association.Trajectory management sets hibernate trajectory cache pool, realizes identity inheritance relying on space-time, position and speed characteristics, repairs broken track by means of sliding window space-time graph optimal matching, and increases collision consistency constraint.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, specifically to a multi-target tracking method and vehicle for autonomous driving scenarios. Background Technology

[0002] With the rapid development of autonomous driving technology, vehicle environmental perception systems need to achieve continuous, stable, and real-time three-dimensional multi-target tracking of surrounding obstacles in complex and dynamic traffic scenarios to obtain their motion states such as position, speed, size, and orientation, providing a basis for trajectory prediction and motion planning. Urban road environments present problems such as mutual occlusion of targets, abrupt changes in motion patterns, sensor noise, and sparse point clouds at long distances, posing severe challenges to the robustness and accuracy of multi-target tracking.

[0003] Existing 3D multi-object tracking methods mostly adopt a "tracking-detection" paradigm. This involves first using a detector to output 3D bounding boxes, then predicting the target's motion state using Kalman filtering, and finally associating the bounding boxes with existing trajectories based on geometric similarity. In the data association stage, a primary matching is typically performed on a bird's-eye view plane using the Hungarian algorithm, followed by secondary matching of unmatched items projected onto the image plane. Trajectory lifecycle management relies on a fixed threshold for missing frames; trajectories that time out are deleted, and unmatched bounding boxes are used to initialize new trajectories.

[0004] However, the aforementioned methods have significant limitations in real-world scenarios. Kalman filtering typically uses a fixed observation noise covariance, failing to adapt to the dynamic changes in noise levels of the detection box with distance, point cloud density, and confidence, leading to severe drift in the state estimation of distant or occluded targets. Data association cost functions often rely on geometry, ignoring motion trends; fixed gating thresholds cannot adaptively adjust based on detection quality; and secondary matching is based solely on projection intersection-union ratio (IoU), failing to fully utilize instance segmentation or point cloud density information, resulting in increased association error rates and frequent identity jumps. Trajectory lifecycle management lacks uncertainty considerations and legitimacy verification, and there is no online reconnection mechanism after trajectory breakage, often assigning multiple identities to the same target, severely impacting tracking continuity and the reliability of downstream tasks. Therefore, a 3D multi-target tracking method that adaptively perceives uncertainty, integrates robust associations from multiple sources, and optimizes trajectory segment consistency online is urgently needed. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a multi-target tracking method for autonomous driving scenarios, comprising the following steps: S1. Acquire continuous frame 3D point cloud and 2D image data of the corresponding tracked target detected by the vehicle-mounted detector, and input them into the fused 3D detector to generate a 3D detection box. S2. Construct the UMRTrack target tracking model. Input the three-dimensional detection box into the UMRTrack model and output the continuous motion trajectory of each target through three stages: state estimation, data association and trajectory management. S3. In the state estimation stage, an online noise adaptive filtering method is adopted to construct the observation noise covariance matrix based on the geometric and semantic information of the detection box and adjust the Kalman gain to suppress state drift caused by long distance and occlusion. S4. In the data association stage, a robust association strategy based on multi-source evidence is adopted. The comprehensive quality score of the detection box is calculated and the Mahalanobis distance threshold is adaptively adjusted. The geometric and motion direction costs are weighted and fused to form a hybrid cost. The main matching of the bird's-eye view is completed through the Hungarian algorithm. For unmatched targets, cross-view projection guided by point cloud density is used to achieve secondary association. S5. In the trajectory management phase, a trajectory fragment graph optimization mechanism is adopted to maintain a dormant trajectory cache pool. Identity inheritance is achieved by combining spatiotemporal proximity, position continuity and velocity consistency. The broken trajectory is recovered by maximum weight matching of the sliding window spatiotemporal graph, and collision consistency constraints are applied at the same time.

[0006] Preferably, step S1 specifically includes the following steps: Collect continuous frames of 3D point cloud data output by the vehicle-mounted LiDAR, denoted as... ,in For point cloud data sets, Total number of frames For the first Frame point cloud collection, The number of points in the point cloud of this frame. Let k be the three-dimensional position coordinates of the k-th point in the point cloud of this frame; Collect continuous frames of two-dimensional image data synchronously output by the vehicle-mounted camera, denoted as... ,in For image data sets, For the first Frame images, the consecutive frame two-dimensional image data being used as input data when the system has a camera; The above three-dimensional point cloud data With two-dimensional image data A pre-trained fusion 3D detector with common input; The fusion 3D detector processes the data of each frame and outputs the 3D detection bounding boxes of all targets in that frame; let the set of 3D detection bounding boxes for consecutive frames be denoted as . ,in, Total number of frames For frame index; the first The set of detection boxes for the frame is ,in For the first The total number of targets detected in the frame. For the first The first frame Each 3D detection bounding box Includes the following information: the spatial coordinates of the target. 3D dimensions Orientation angle Detection confidence and the number of point clouds within the frame This provides input for subsequent tracking.

[0007] Preferably, step S2 specifically includes the following steps: Input the set of 3D detection boxes for the current frame. Each detection box Includes location ,size Orientation Angle Detection confidence and the number of point clouds within the frame Simultaneously input the trajectory set of the previous frame. , For the first The first frame There are 3 trajectories, each containing a state vector. Covariance Matrix ; Detection and trajectory preprocessing are performed on each detection box in the current frame. Extract center distance Point cloud density Detection confidence and the standard deviation of detection noise This forms the detection feature vector for each trajectory in the previous frame. Extract the previous frame status Previous Frame covariance trajectory velocity Number of consecutive unmatched frames and trajectory reliability , which constitute the trajectory feature vector; For the previous frame Each trajectory State prediction is performed using decoupled Kalman filtering to obtain the current frame. state and the current frame covariance Simultaneously, the adaptive observation noise covariance matrix is ​​calculated online based on the detection characteristics. And substitute it into the Kalman filter update formula to dynamically adjust the Kalman gain. ; Trajectory and detection association matching is performed sequentially on the bird's-eye view plane and the image plane; the first matching is performed on the bird's-eye view plane: the overall quality score of each detection box is calculated. Implement three-level adaptive gating and utilize hybrid cost Construct the cost matrix, where For the cost of geometric consistency, The cost of maintaining consistency in the direction of motion, and The corresponding weights are assigned; the first matching is performed using the Hungarian algorithm; the second matching is performed on the image plane: for unmatched trajectories and detections on the bird's-eye view plane, the 3D detection boxes are projected onto the image plane, and density similarity is calculated. A distance-adaptive density threshold is used to construct the quadratic matching cost. And the association is completed through a greedy algorithm; Perform trajectory initialization, updating, and termination. For successfully matched trajectories, use the position parameters of the detection boxes that matched the trajectory. Perform Kalman state updates; initialize new trajectories using unmatched detection boxes. When a new trajectory matches a dormant segment in the cache pool, it inherits the original identity and smoothly injects the splicing point velocity; for trajectories with consecutive unmatched frames and confidence levels below the threshold, save the last 10 frames of their state as dormant segments and terminate the trajectory; trigger maximum weight matching of the spatiotemporal graph with a sliding window of 50 frames every 10 frames to reconnect broken segments; apply collision consistency constraints to the output trajectory. Output trajectory status, output updated trajectory set Each track contains a unique identifier. and the continuous motion state of the current frame: position ,size ,speed acceleration Orientation Angle and trajectory credibility .

[0008] Preferably, step S3 specifically includes the following steps: establishing a target position filter, a size filter, and an orientation angle filter, respectively, with each filter independently performing Kalman filtering state estimation; in the position filter, its state vector is... ,in This is the current frame state vector. It indicates the fusion of the first Posterior estimated state after the frame detection box Let the center coordinates of the target be on the bird's-eye view plane. For velocity components, For acceleration components; the prediction step uses a constant acceleration model:

[0009]

[0010] in, This is the state vector from the previous frame. This indicates that the previous frame was fused with the first frame. The posterior estimated state of the frame detection box. This is the state prediction vector. Indicates the use of the first Frame trajectory to predict the first The prior prediction state of the frame trajectory, i.e., before the use of the first... Frame detection box, Here is the state transition matrix. The process noise covariance matrix is ​​preset. To predict the covariance matrix, Estimate the covariance matrix for the previous frame; For each detection box in the current frame, based on its center coordinates ,size Detection confidence and the number of point clouds within the frame Online calculation of adaptive observation noise variance; specifically, firstly, the center distance is calculated. Point cloud density Then construct the distance-related noise term:

[0011]

[0012] in Coordinates of distance from center The corresponding variance of the noise. Distance noise figure; Point cloud density-related noise term:

[0013] in For the variance of density-related noise, Density noise figure, Reference density; detection confidence-related noise term:

[0014] in The variance of confidence level related noise. The confidence level noise figure; Based on the base noise figure, the total observed noise variance in the direction is then obtained:

[0015]

[0016] Construct the adaptive observation noise covariance matrix:

[0017]

[0018] When a detection box successfully matches a trajectory, Substituting into the Kalman filter update formula, we first calculate the innovation covariance:

[0019] in To innovate covariance, The observation matrix is ​​given; then the Kalman gain is calculated:

[0020] Finally, the observations from the current frame are used. Perform a status update:

[0021]

[0022] The size filter and orientation angle filter are updated in exactly the same way as the position filter, only by replacing their state vector and observation matrix with the corresponding dimensions and adjusting the noise parameters to the dimensions of the corresponding physical quantities.

[0023] Preferably, step S4 specifically includes the following steps: First, calculate the detection box for each detection box in the current frame. Overall quality score This score is determined by the detection confidence level. Distance factor Point cloud density factor and detection of noise factors Multiplying the four together, we get:

[0024] in The distance between the centers of the detection frames. This is the maximum distance at which the sensor can reliably detect data. Point cloud density, Here is the reference density threshold, where The standard deviation of the detection noise estimated online. According to the maximum permissible noise standard, The values ​​categorize the detection boxes into three levels: High level: Medium level: And lower levels: A three-level verification and filtering system is implemented: high-level detection boxes proceed directly to subsequent matching; medium-level detection boxes must meet the Mahalanobis distance requirement. ,in The Mahalanobis distance threshold is set; low-level detection boxes are only detected if the spatial Euclidean distance to the trajectory is less than 5 meters and the number of consecutive unmatched frames of the trajectory is [not specified]. Passed in time; For the verified trajectory and detection pair, construct a hybrid cost matrix in the bird's-eye view plane; For each trajectory and each detection box First calculate the geometric consistency cost. Cost of consistency with direction of motion The geometric cost is calculated based on the normalized Ro_GDIoU, where Ro_GDIoU is the generalized distance intersection-union ratio of the rotated bounding box, which is a matching metric that combines rotation IoU, center distance, and the length of the diagonal of the circumscribed rectangle.

[0025]

[0026] in, The area of ​​the intersection of the two detection boxes and the trajectory box. For the area of ​​the union, Let the area of ​​the smallest bounding rectangle of the two boxes be the area of ​​the bounding rectangle. The Euclidean distance between the center points of the two frames. The distance between the diagonals of the smallest bounding rectangle of the two boxes. Weights set; Consistency cost of motion direction Location prediction based on trajectory Center of the detection frame displacement vector and trajectory speed Cosine similarity:

[0027]

[0028] The combined cost is the weighted sum of the two:

[0029] Let geometric consistency weight Consistency weight of motion direction ; Based on the above hybrid cost matrix, a fixed matching threshold is set. The Hungarian algorithm is used to complete the main matching on the bird's-eye view plane; For the unmatched targets remaining after the main matching, including trajectories and detection boxes, a cross-view secondary matching is performed. First, the eight corner points of the unmatched 3D detection boxes or trajectory prediction boxes are projected onto the image plane to obtain 2D bounding boxes. The system provides a 2D instance segmentation mask and calculates the number of points in the point cloud belonging to the same instance mask within the projection frame. Total number of points within the 3D frame The ratio of density similarity :

[0030] Using a fixed density threshold Combining geometric similarity SDIoU and density similarity, a quadratic matching cost is constructed:

[0031] Finally, a greedy algorithm is used to complete the association, according to... Sort from highest to lowest, and select the matching trajectory for each unmatched trajectory in turn. and The best detection box is removed from the candidate set after a successful match.

[0032] Preferably, step S5 specifically includes the following steps: When a trajectory In frame Credibility Below the preset deletion threshold When the trajectory has terminated, it is determined that the trajectory has been terminated; when a new trajectory is initialized by an unmatched detection box in the current frame, it is compared with the cache pool. Match all dormant segments; the matching condition requires that the following four constraints be met simultaneously: time difference Seconds, Euclidean distance between the first and last positions Meters, the angle between the velocity direction of the first frame of the new trajectory and the velocity direction of the last frame of the dormant segment. And both must be of the same category; if a match is found, the new trajectory inherits the identity identifier of the dormant segment. The Kalman filter state is initialized as follows: position is taken from newly detected observations, and velocity is injected using weighted smoothing.

[0033] in The final velocity of the dormant segment. The speed calculated from the first frame detection of the new trajectory. The weighting coefficients are set to control the fusion ratio between the instantaneous velocity calculated by the new detection box and the final velocity of the historical trajectory before dormancy; the covariance matrix is ​​initialized to... To reflect the uncertainty of forecasting.

[0034] Preferably, step S5 further includes the following step: to repair internal track breaks caused by short-term occlusion or detection noise, the system needs to maintain a length of A sliding window of frames, which uses the current frame as the frame. The right endpoint is included, and the past is also included. All the information in a frame, all active trajectories within the window, and the dormant segments together constitute a local spatiotemporal graph, denoted as . The system uses a set of nodes (nodes, edges). Each node in the node set (node∈Nodes) represents the specific existence state of a trajectory or segment within a frame of the window. Each edge in the edge set (edge∈Edges) connects two different nodes belonging to the same physical target, indicating that these two states are related in time and space. To control the complexity of the graph and ensure real-time performance, candidate edges are only established between two nodes when the difference between their corresponding frame indices does not exceed 5 frames. Subsequently, every 10 frames processed, the system performs a global maximum weight matching algorithm on the local spatiotemporal graph. Successfully matched node pairs are considered as different segments of the same physical target and are then merged in chronological order. The identity ID is uniformly retained as the earlier created ID to ensure the long-term stability of the label.

[0035] Preferably, step S5 further includes the following steps: to maintain the physical rationality of the output trajectory, perform pairwise collision detection on all confirmed trajectories within the current window; for any two trajectories, if there exists a frame such that their 3D bounding boxes intersect (comparison and union) ratio... If an unacceptable penetration occurs, the intersection-union ratio (IUR) of the 3D bounding box is calculated using the following formula: ,in , These are the volumes of the two 3D bounding boxes. The volume of the overlapping area of ​​the two boxes; compare the confidence levels of the two trajectories in the current frame. and Select the one with lower confidence level as the adjustment target; calculate the overlap depth along the line connecting the centers of the two frames. The selected trajectory is translated outward along the line connecting the centers of the two frames. The distance; the outward translation The distance only affects the output trajectory and does not feed back to the internal state of the Kalman filter, thus avoiding causal violations; when multiple trajectories overlap, they are adjusted sequentially from low to high confidence; finally, all output trajectories satisfy the physical constraint of non-penetration, and the identity is effectively maintained after long-term occlusion.

[0036] An automobile includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement the multi-target tracking method for autonomous driving scenarios as described above.

[0037] By adopting the above scheme, this invention has the following advantages and beneficial effects: This invention proposes a robust 3D multi-target tracking method for uncertainty perception in autonomous driving, building a complete perception closed loop from 3D detection boxes to motion trajectories, and optimizing it around three major modules: state estimation, data association, and trajectory management. The state estimation module introduces online noise adaptive Kalman filtering, combining target distance, point cloud density, and detection confidence to solve the observation noise covariance matrix in real time, dynamically adjusting the Kalman gain to suppress state drift caused by long distance and target occlusion; at the same time, it defines trajectory credibility based on posterior covariance and the number of unmatched frames, realizing refined management of the trajectory lifecycle. The data association module adopts a robust association strategy based on multi-source evidence: it sets a three-level adaptive gating based on the comprehensive quality score to reduce ghost trajectories; it integrates geometric and motion direction costs to construct a hybrid cost matrix, dynamically adjusting weights and matching thresholds to adapt to nonlinear motion scenarios such as sharp turns and speed changes; and it combines 2D instance segmentation masks to calculate density similarity, completing cross-view secondary matching and improving matching accuracy under depth deviation. The trajectory management module introduces trajectory segment graph optimization, which saves terminated trajectories as dormant segments, and new trajectories can be matched and inherited from the original ID and have their speed smoothed; broken trajectories are merged by maximum weight matching of the spatiotemporal graph through a sliding window; collision consistency constraints are added to perform translation correction on overlapping trajectories to ensure that the output trajectory conforms to physical laws. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the three-dimensional multi-target tracking method of the present invention.

[0039] Figure 2 This is a flowchart of the UMRTrack framework of the present invention.

[0040] Figure 3 This is a flowchart illustrating the robust association strategy for multi-source evidence of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Reference manual attached Figure 1-3 The first embodiment of the present invention provides a multi-target tracking method for autonomous driving scenarios, which can be executed by a vehicle. Specifically, it is executed by one or more processors in the vehicle to implement steps S1 to S5.

[0043] S1. Acquire continuous frame 3D point cloud and 2D image data of the corresponding tracked target detected by the vehicle-mounted detector, and input them into the fused 3D detector to generate a 3D detection box. S1 includes steps S11 to S14.

[0044] S11. Acquire continuous frames of 3D point cloud data output by the vehicle-mounted lidar, denoted as... ,in For point cloud data sets, Total number of frames For the first Frame point cloud collection, The number of points in the point cloud of this frame. Let k be the three-dimensional position coordinates of the k-th point in the point cloud of this frame; S12. Acquire continuous frame two-dimensional image data synchronously output by the vehicle-mounted camera, denoted as... ,in For image data sets, For the first Frame images, the consecutive frame two-dimensional image data being used as input data when the system has a camera; S13. Transfer the above three-dimensional point cloud data With two-dimensional image data A pre-trained fusion 3D detector with common input; S14. The fused 3D detector processes the data of each frame and outputs the 3D detection bounding boxes of all targets in that frame; let the set of 3D detection bounding boxes for consecutive frames be denoted as . ,in, Total number of frames For frame index; the first The set of detection boxes for the frame is ,in For the first The total number of targets detected in the frame. For the first The first frame Each 3D detection bounding box Includes the following information: the spatial coordinates of the target. 3D dimensions Orientation Angle Detection confidence and the number of point clouds within the frame This provides input for subsequent tracking.

[0045] S2. Construct the UMRTrack target tracking model. Input the three-dimensional detection box into the UMRTrack model and output the continuous motion trajectory of each target through three stages: state estimation, data association and trajectory management. Preferably, S2 includes the following steps: S21. Input the set of 3D detection boxes for the current frame. Each detection box Includes location ,size Orientation Angle Detection confidence and the number of point clouds within the frame Simultaneously input the trajectory set of the previous frame. , For the first The first frame There are 3 trajectories, each containing a state vector. Covariance Matrix ; S22. Perform detection and trajectory preprocessing for each detection box in the current frame. Extract center distance Point cloud density Detection confidence and the standard deviation of detection noise This forms the detection feature vector for each trajectory in the previous frame. Extract the previous frame status Previous Frame covariance trajectory velocity Number of consecutive unmatched frames and trajectory reliability , which constitute the trajectory feature vector; S23, Regarding the previous frame Each trajectory State prediction is performed using decoupled Kalman filtering to obtain the current frame. state and the current frame covariance Simultaneously, the adaptive observation noise covariance matrix is ​​calculated online based on the detection characteristics. And substitute it into the Kalman filter update formula to dynamically adjust the Kalman gain. ; S24. Perform trajectory and detection association matching, sequentially on the bird's-eye view plane and the image plane; perform the first matching on the bird's-eye view plane: calculate the overall quality score for each detection box. Implement three-level adaptive gating and utilize hybrid cost Construct the cost matrix, where For the cost of geometric consistency, The cost of maintaining consistency in the direction of motion, and The corresponding weights are assigned; the first matching is performed using the Hungarian algorithm; the second matching is performed on the image plane: for unmatched trajectories and detections on the bird's-eye view plane, the 3D detection boxes are projected onto the image plane, and density similarity is calculated. A distance-adaptive density threshold is used to construct the quadratic matching cost. The association is completed through a greedy algorithm (see the literature for the Hungarian algorithm and the greedy algorithm: Chen Zhichao, Chang Jiayun, Xu Zhenyu, et al. Multi-target tracking algorithm based on SORT framework unscented Kalman filter [J]. Guidance and Fuze, 2025, 46(4):7-14,47.). S25. Perform trajectory initialization, update, and termination. For successfully matched trajectories, use the position parameters of the detection boxes that successfully match the trajectory. Perform Kalman state updates; initialize new trajectories using unmatched detection boxes. When a new trajectory matches a dormant segment in the cache pool, it inherits the original identity and smoothly injects the splicing point velocity; for trajectories with consecutive unmatched frames and confidence levels below the threshold, save the last 10 frames of their state as dormant segments and terminate the trajectory; trigger maximum weight matching of the spatiotemporal graph with a sliding window of 50 frames every 10 frames to reconnect broken segments; apply collision consistency constraints to the output trajectory. S26. Output trajectory status, output the updated trajectory set. Each track contains a unique identifier. and the continuous motion state of the current frame: position ,size ,speed acceleration Orientation Angle and trajectory credibility .

[0046] S3. In the state estimation stage, an online noise adaptive filtering method is adopted to construct the observation noise covariance matrix based on the geometric and semantic information of the detection box and adjust the Kalman gain to suppress state drift caused by long distance and occlusion. Preferably, S3 includes the following steps: S31. Establish the target's position filter, size filter, and orientation angle filter respectively, and perform Kalman filtering state estimation independently for each filter; in the position filter, its state vector is... ,in This is the current frame state vector. It indicates the fusion of the first Posterior estimated state after the frame detection box Let the center coordinates of the target be on the bird's-eye view plane. For velocity components, For acceleration components; the prediction step uses a constant acceleration model:

[0047]

[0048] in, This is the state vector from the previous frame. This indicates that the previous frame was fused with the first frame. The posterior estimated state of the frame detection box. This is the state prediction vector. Indicates the use of the first Frame trajectory to predict the first The prior prediction state of the frame trajectory, i.e., before the use of the first... Frame detection box, Here is the state transition matrix. The process noise covariance matrix is ​​preset. To predict the covariance matrix, Estimate the covariance matrix for the previous frame; S32. For each detection box in the current frame, based on its center coordinates... ,size Detection confidence and the number of point clouds within the frame Online calculation of adaptive observation noise variance; specifically, firstly, the center distance is calculated. Point cloud density Then construct the distance-related noise term:

[0049]

[0050] in Coordinates of distance from center The corresponding variance of the noise. Distance noise figure; Point cloud density-related noise term:

[0051] in For the variance of density-related noise, Density noise figure, Reference density; detection confidence-related noise term:

[0052] in The variance of confidence level related noise. The confidence level noise figure; Based on the base noise figure, the total observed noise variance in the direction is then obtained:

[0053]

[0054] Construct the adaptive observation noise covariance matrix:

[0055]

[0056] S33. When a detection box successfully matches a trajectory, Substituting into the Kalman filter update formula, we first calculate the innovation covariance:

[0057] in To innovate covariance, The observation matrix is ​​given; then the Kalman gain is calculated:

[0058] Finally, the observations from the current frame are used. Perform a status update:

[0059]

[0060] The update methods for S34, size filter, and orientation angle filter are exactly the same as those for position filter. Only the state vector and observation matrix need to be replaced with the corresponding dimensions, and the noise parameters need to be adjusted to the dimensions of the corresponding physical quantities.

[0061] Through the above steps, when the detection box has significant noise due to long distance, sparse point cloud, or low confidence, Automatically increases the Kalman gain. This reduces the impact of unreliable observations on state estimation and effectively suppresses state drift caused by long distances and occlusion.

[0062] For a size filter, its state vector is ,in For the target length, For target width, , These represent the rates of change for length and width, respectively. The observation matrix is... Used to extract the size observation value of the current frame. Observation noise covariance matrix The construction method is similar to that of a position filter, but the dimensions and amplitudes of each noise component need to be adjusted: reference noise Pick Distance noise figure Pick Density noise amplitude Pick Reference density Still Confidence noise amplitude Pick In the distance-related noise term, since size changes are mainly affected by the target's projection onto the image plane, the distance to the center of the detection box can be considered... The radial projection square is replaced with and The weighted form can also be simplified to direct use. No direction decomposition. With the above settings, when the detection frame experiences significant noise in size observation due to distance or sparse point cloud conditions, Automatic increase, gain This reduces the impact of unreliable observations on size estimation.

[0063] For the orientation angle filter, its state vector is ,in For the target yaw angle, This represents the rate of change of the yaw angle. The observation matrix is... The observed value is Observation noise covariance matrix During construction, reference noise Pick Distance noise figure Pick (Unit: rad / m), distance-related noise term is Density noise amplitude Pick Confidence level noise amplitude Pick Similarly, the exponential decay model is used. .Will By substituting the innovative covariance calculation, the gain decreases as noise increases, thus ensuring the robustness of the orientation angle estimation.

[0064] By independently implementing the size filter and orientation angle filter, this method achieves stable and adaptive estimation of the target's three-dimensional dimensions and yaw angle, forming a complete decoupled Kalman filter system together with the position filter. When the detection box has significant noise due to long distance, sparse point cloud, or low confidence, the effective observation noise covariance of each filter... , All of these automatically increase, resulting in the corresponding Kalman gain. , This reduces the impact of unreliable observations on state estimation and effectively suppresses state drift caused by long distances and occlusion.

[0065] S4. In the data association stage, a robust association strategy based on multi-source evidence is adopted. The comprehensive quality score of the detection box is calculated and the Mahalanobis distance threshold is adaptively adjusted. The geometric and motion direction costs are weighted and fused to form a hybrid cost. The main matching of the bird's-eye view is completed through the Hungarian algorithm. For unmatched targets, cross-view projection guided by point cloud density is used to achieve secondary association. Preferably, S4 includes the following steps: S41. First, calculate each detection box in the current frame. Overall quality score This score is determined by the detection confidence level. Distance factor Point cloud density factor and detection of noise factors Multiplying the four together, we get:

[0066] in The distance between the centers of the detection frames. This is the maximum distance at which the sensor can reliably detect data. Point cloud density, Here is the reference density threshold, where The standard deviation of the detection noise estimated online. According to the maximum permissible noise standard, The values ​​categorize the detection boxes into three levels: High level: Medium level: And lower levels: A three-level verification and filtering system is implemented: high-level detection boxes proceed directly to subsequent matching; medium-level detection boxes must meet the Mahalanobis distance requirement. ,in The Mahalanobis distance threshold is set; low-level detection boxes are only detected if the spatial Euclidean distance to the trajectory is less than 5 meters and the number of consecutive unmatched frames of the trajectory is [not specified]. Passed in time; S42. For the verified trajectories and detection pairs, construct a hybrid cost matrix in the bird's-eye view plane; for each trajectory... and each detection box First calculate the geometric consistency cost. Cost of consistency with direction of motion The geometric cost is calculated based on the normalized Ro_GDIoU, where Ro_GDIoU is the generalized distance intersection-union ratio of the rotated bounding box, which is a matching metric that combines rotation IoU, center distance, and the length of the diagonal of the circumscribed rectangle.

[0067]

[0068] in, The area of ​​the intersection of the two detection boxes and the trajectory box. For the area of ​​the union, Let the area of ​​the smallest bounding rectangle of the two boxes be the area of ​​the bounding rectangle. The Euclidean distance between the center points of the two frames. The distance between the diagonals of the smallest bounding rectangle of the two boxes. Weights set; Consistency cost of motion direction Location prediction based on trajectory Center of the detection frame displacement vector and trajectory speed Cosine similarity:

[0069]

[0070] The combined cost is the weighted sum of the two:

[0071] Let geometric consistency weight Consistency weight of motion direction ; S43. Based on the above hybrid cost matrix, set a fixed matching threshold. The Hungarian algorithm is used to complete the main matching on the bird's-eye view plane, and the output of the Hungarian algorithm is used as the main matching result of the bird's-eye view plane.

[0072] S44. For the remaining unmatched targets after the main matching, including trajectories and detection boxes, perform cross-view secondary matching. First, project the 8 corner points of the unmatched 3D detection boxes or trajectory prediction boxes onto the image plane to obtain 2D bounding boxes.

[0073] The system provides a 2D instance segmentation mask and calculates the number of points in the point cloud belonging to the same instance mask within the projection frame. Total number of points within the 3D frame The ratio of density similarity :

[0074] Using a fixed density threshold Combining geometric similarity SDIoU and density similarity, a quadratic matching cost is constructed:

[0075] Finally, a greedy algorithm is used to complete the association, according to... Sort from highest to lowest, and select the matching trajectory for each unmatched trajectory in turn. and The best detection box is removed from the candidate set after a successful match.

[0076] S5. In the trajectory management phase, a trajectory fragment graph optimization mechanism is adopted to maintain a dormant trajectory cache pool. Identity inheritance is achieved by combining spatiotemporal proximity, position continuity and velocity consistency. The broken trajectory is recovered by maximum weight matching of the sliding window spatiotemporal graph, and collision consistency constraints are applied at the same time.

[0077] Preferably, S5 includes the following steps: S51, When a trajectory In frame Credibility Below the preset deletion threshold At that time, it was determined that the trajectory had terminated; S52. When a new trajectory is initialized by an unmatched detection box in the current frame, it is compared with the buffer pool. Match all dormant segments; the matching condition requires that the following four constraints be met simultaneously: time difference Seconds, Euclidean distance between the first and last positions Meters, the angle between the velocity direction of the first frame of the new trajectory and the velocity direction of the last frame of the dormant segment. And both must be of the same category; if a match is found, the new trajectory inherits the identity identifier of the dormant segment. The Kalman filter state is initialized as follows: position is taken from newly detected observations, and velocity is injected using weighted smoothing.

[0078] in The final velocity of the dormant segment. The speed calculated from the first frame detection of the new trajectory. The weighting coefficients are set to control the fusion ratio between the instantaneous velocity calculated by the new detection box and the final velocity of the historical trajectory before dormancy; the covariance matrix is ​​initialized to... To reflect the uncertainty of forecasting.

[0079] S53. To repair internal track breaks caused by short-term occlusion or detection noise, the system needs to maintain a length of... A sliding window of frames, which uses the current frame as the frame. The right endpoint is included, and the past is also included. All the information in a frame, all active trajectories within the window, and the dormant segments together constitute a local spatiotemporal graph, denoted as . Nodes, Edges); where each node in the node set Nodes represents the specific existence state of a trajectory or segment in a certain frame within the window, and each edge in the edge set Edges is used to connect two different nodes belonging to the same physical target, indicating that the two states are related in time and space; in order to control the complexity of the graph and ensure real-time performance, a candidate edge is established between two nodes only when the difference between the frame indices corresponding to the two nodes does not exceed 5 frames. After that, every 10 frames processed, the system performs a global maximum weight matching algorithm on the subgraph. The successfully matched node pairs are regarded as different segments of the same physical target and then merged in chronological order; the identity ID is uniformly retained as the earlier created ID to ensure the long-term stability of the label (for the maximum weight matching algorithm, see the literature: Yao Rugui, Song Doudou, Yao Lukun, et al. A fair matching method based on maximum weight matching [P]. CN111935798A [P]. 2020-11-13.).

[0080] S54. To maintain the physical plausibility of the output trajectory, perform pairwise collision detection on all confirmed trajectories within the current window; for any two trajectories, if there exists a frame such that their 3D bounding boxes intersect (comparison and union) ratio is... If an unacceptable penetration occurs, the intersection-union ratio (IUR) of the 3D bounding box is calculated using the following formula: ,in , These are the volumes of the two 3D bounding boxes. The volume of the overlapping area of ​​the two boxes; compare the confidence levels of the two trajectories in the current frame. and Select the one with lower confidence level as the adjustment target; calculate the overlap depth along the line connecting the centers of the two frames. The selected trajectory will be translated outward along this direction. The distance; the translation only acts on the output trajectory and does not feed back to the internal state of the Kalman filter, thus avoiding causal violations; when multiple trajectories overlap, they are adjusted in order of confidence from low to high; finally, all output trajectories satisfy the physical constraint of non-penetration, and the identity is effectively maintained after long-term occlusion.

[0081] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-target tracking method for autonomous driving scenarios, characterized in that, Includes the following steps: S1. Acquire continuous frame 3D point cloud and 2D image data of the corresponding tracked target detected by the vehicle-mounted detector, and input them into the fused 3D detector to generate a 3D detection box. S2. Construct the UMRTrack target tracking model. Input the 3D detection boxes into the UMRTrack model, and sequentially perform three stages: state estimation, data association, and trajectory management, to output the continuous motion trajectory of each target. Specifically, input the set of 3D detection boxes for the current frame. Each detection box Includes location ,size Orientation angle Detection confidence and the number of point clouds within the frame Simultaneously input the trajectory set of the previous frame. , For the first The first frame There are 3 trajectories, each containing a state vector. Covariance Matrix ; Detection and trajectory preprocessing are performed on each detection box in the current frame. Extract center distance Point cloud density Detection confidence and the standard deviation of detection noise This forms the detection feature vector for each trajectory in the previous frame. Extract the previous frame status Previous Frame covariance trajectory velocity Number of consecutive unmatched frames and trajectory reliability , which constitute the trajectory feature vector; For the previous frame Each trajectory State prediction is performed using decoupled Kalman filtering to obtain the current frame. state and the current frame covariance Simultaneously, the adaptive observation noise covariance matrix is ​​calculated online based on the detection characteristics. And substitute it into the Kalman filter update formula to dynamically adjust the Kalman gain. ; Trajectory and detection association matching is performed sequentially on the bird's-eye view plane and the image plane; the first matching is performed on the bird's-eye view plane: the overall quality score of each detection box is calculated. Implement three-level adaptive gating and utilize hybrid cost Construct the cost matrix, where For the cost of geometric consistency, The cost of maintaining consistency in the direction of motion, and The corresponding weights are assigned; the first matching is performed using the Hungarian algorithm; the second matching is performed on the image plane: for unmatched trajectories and detections on the bird's-eye view plane, the 3D detection boxes are projected onto the image plane, and density similarity is calculated. A distance-adaptive density threshold is used to construct the quadratic matching cost. And the association is completed through a greedy algorithm; Perform trajectory initialization, updating, and termination. For successfully matched trajectories, use the position parameters of the detection boxes that matched the trajectory. Perform Kalman state updates; initialize new trajectories using unmatched detection boxes. When a new trajectory matches a dormant segment in the cache pool, it inherits the original identity and smoothly injects the splicing point velocity; for trajectories with consecutive unmatched frames and confidence levels below the threshold, save the last 10 frames of their state as dormant segments and terminate the trajectory; trigger maximum weight matching of the spatiotemporal graph with a sliding window of 50 frames every 10 frames to reconnect broken segments; apply collision consistency constraints to the output trajectory. Output trajectory status, output updated trajectory set Each track contains a unique identifier. and the continuous motion state of the current frame: position ,size ,speed acceleration Orientation angle and trajectory credibility ; S3. In the state estimation stage, an online noise adaptive filtering method is adopted to construct the observation noise covariance matrix based on the geometric and semantic information of the detection box and adjust the Kalman gain to suppress state drift caused by long distance and occlusion. S4. In the data association stage, a robust association strategy based on multi-source evidence is adopted. The comprehensive quality score of the detection box is calculated and the Mahalanobis distance threshold is adaptively adjusted. The geometric and motion direction costs are weighted and fused to form a hybrid cost. The main matching of the bird's-eye view is completed through the Hungarian algorithm. For unmatched targets, cross-view projection guided by point cloud density is used to achieve secondary association. S5. In the trajectory management phase, a trajectory fragment graph optimization mechanism is adopted to maintain a dormant trajectory cache pool. Identity inheritance is achieved by combining spatiotemporal proximity, position continuity and velocity consistency. The broken trajectory is recovered by maximum weight matching of the sliding window spatiotemporal graph, and collision consistency constraints are applied at the same time.

2. The multi-target tracking method for autonomous driving scenarios according to claim 1, characterized in that, S1 specifically includes the following steps: Collect continuous frames of 3D point cloud data output by the vehicle-mounted LiDAR, denoted as... ,in For point cloud data sets, Total number of frames For the first Frame point cloud collection, The number of points in the point cloud of this frame. Let k be the three-dimensional position coordinates of the k-th point in the point cloud of this frame; Collect continuous frames of two-dimensional image data synchronously output by the vehicle-mounted camera, denoted as... ,in For image data sets, For the first Frame images, the consecutive frame two-dimensional image data being used as input data when the system has a camera; The above three-dimensional point cloud data With two-dimensional image data A pre-trained fusion 3D detector with common input; The fusion 3D detector processes the data of each frame and outputs the 3D detection bounding boxes of all targets in that frame; let the set of 3D detection bounding boxes for consecutive frames be denoted as . ,in, Total number of frames For frame index; the first The set of detection boxes for the frame is ,in For the first The total number of targets detected in the frame. For the first The first frame Each 3D detection bounding box Includes the following information: the spatial coordinates of the target. 3D dimensions Orientation angle Detection confidence and the number of point clouds within the frame This provides input for subsequent tracking.

3. The multi-target tracking method for autonomous driving scenarios according to claim 1, characterized in that, S3 specifically includes the following steps: establishing a target position filter, a size filter, and an orientation angle filter, respectively; each filter independently performs Kalman filtering state estimation; in the position filter, its state vector is... ,in This is the current frame state vector. It indicates the fusion of the first Posterior estimated state after the frame detection box Let the center coordinates of the target be on the bird's-eye view plane. For velocity components, For acceleration components; the prediction step uses a constant acceleration model: in, This is the state vector from the previous frame. This indicates that the previous frame was merged with the first frame. The posterior estimated state of the frame detection box. This is the state prediction vector. Indicates the use of the first Frame trajectory to predict the first The prior prediction state of the frame trajectory, i.e., before the use of the first... Frame detection box, Here is the state transition matrix. The process noise covariance matrix is ​​preset. To predict the covariance matrix, Estimate the covariance matrix for the previous frame; For each detection box in the current frame, based on its center coordinates ,size Detection confidence and the number of point clouds within the frame Online calculation of adaptive observation noise variance; specifically, firstly, the center distance is calculated. Point cloud density Then construct the distance-dependent noise term: in Coordinates of distance from center The corresponding variance of the noise. Distance noise figure; Point cloud density-related noise term: in For the variance of density-related noise, Density noise figure, Reference density; detection confidence-related noise term: in The variance of confidence level related noise. The confidence level noise figure; Based on the base noise figure, the total observed noise variance in the direction is then obtained: Construct the adaptive observation noise covariance matrix: When a detection box successfully matches a trajectory, Substituting into the Kalman filter update formula, we first calculate the innovation covariance: in To innovate covariance, The observation matrix is ​​given; then the Kalman gain is calculated: Finally, the observations from the current frame are used. Perform a status update: The size filter and orientation angle filter are updated in exactly the same way as the position filter, only by replacing their state vector and observation matrix with the corresponding dimensions and adjusting the noise parameters to the dimensions of the corresponding physical quantities.

4. The multi-target tracking method for autonomous driving scenarios according to claim 1, characterized in that, S4 specifically includes the following steps: First, calculate the detection box for each detection box in the current frame. Overall quality score This score is determined by the detection confidence level. Distance factor Point cloud density factor and detection noise factor Multiplying the four together, we get: in The distance between the centers of the detection frames. This is the maximum distance at which the sensor can reliably detect data. Point cloud density, Here is the reference density threshold, where The standard deviation of the detection noise estimated online. According to the maximum permissible noise standard, The values ​​categorize the detection boxes into three levels: High level: Medium level: And lower levels: A three-level verification and filtering system is implemented: high-level detection boxes proceed directly to subsequent matching; medium-level detection boxes must meet the Mahalanobis distance requirement. ,in The set Mahalanobis distance threshold; Low-level detection boxes are only considered if the spatial Euclidean distance between the detection box and the trajectory is less than 5 meters and the number of consecutive unmatched frames of the trajectory is [not specified]. Passed in time; For the verified trajectory and detection pair, construct a hybrid cost matrix in the bird's-eye view plane; For each trajectory and each detection box First calculate the geometric consistency cost. Cost of consistency with direction of motion The geometric cost is calculated based on the normalized Ro_GDIoU, where Ro_GDIoU is the generalized distance intersection-union ratio of the rotated bounding box, which is a matching metric that combines rotation IoU, center distance, and the length of the diagonal of the circumscribed rectangle. in, The area of ​​the intersection of the two detection boxes and the trajectory box. For the area of ​​the union, Let the area of ​​the smallest bounding rectangle of the two boxes be the area of ​​the bounding rectangle. The Euclidean distance between the center points of the two frames is given. The distance between the diagonals of the smallest bounding rectangle of the two boxes. The weights set; Cost of consistent motion direction Location prediction based on trajectory Center of the detection frame displacement vector and trajectory speed Cosine similarity: The combined cost is the weighted sum of the two: Let geometric consistency weight Consistency weight of motion direction ; Based on the above hybrid cost matrix, a fixed matching threshold is set. The Hungarian algorithm is used to complete the main matching on the bird's-eye view plane; For the unmatched targets remaining after the main matching, including trajectories and detection boxes, a cross-view secondary matching is performed. First, the eight corner points of the unmatched 3D detection boxes or trajectory prediction boxes are projected onto the image plane to obtain 2D bounding boxes. The system provides a 2D instance segmentation mask and calculates the number of points in the point cloud belonging to the same instance mask within the projection frame. Total number of points within the 3D frame The ratio of density similarity : Using a fixed density threshold Combining geometric similarity SDIoU and density similarity, a quadratic matching cost is constructed: Finally, a greedy algorithm is used to complete the association, according to... Sort from highest to lowest, and select the matching trajectory for each unmatched trajectory in turn. and The best detection box is removed from the candidate set after a successful match.

5. The multi-target tracking method for autonomous driving scenarios according to claim 1, characterized in that, S5 specifically includes the following steps: When a trajectory In frame Credibility Below the preset deletion threshold When the trajectory has terminated, it is determined that the trajectory has been terminated; when a new trajectory is initialized by an unmatched detection box in the current frame, it is compared with the cache pool. Match all dormant segments; the matching condition requires that the following four constraints be met simultaneously: time difference Seconds, Euclidean distance between the first and last positions Meters, the angle between the velocity direction of the first frame of the new trajectory and the velocity direction of the last frame of the dormant segment. And both must be of the same category; if a match is found, the new trajectory inherits the identity identifier of the dormant segment. The Kalman filter state is initialized as follows: position is taken from newly detected observations, and velocity is injected using weighted smoothing. in The final velocity of the dormant segment. The speed calculated from the first frame detection of the new trajectory. The weighting coefficients are set to control the fusion ratio between the instantaneous velocity calculated by the new detection box and the final velocity of the historical trajectory before dormancy; the covariance matrix is ​​initialized to... To reflect the uncertainty of forecasting.

6. The multi-target tracking method for autonomous driving scenarios according to claim 5, characterized in that, S5 specifically includes the following steps: To repair internal track breaks caused by short-term occlusion or detection noise, the system needs to maintain a length of [length missing]. A sliding window of frames, which uses the current frame as the frame. The right endpoint is included, and the past is also included. All the information in a frame, all active trajectories within the window, and sleep segments together constitute a local spatiotemporal graph, denoted as . The system uses a set of nodes (nodes, edges). Each node in the node set (node∈Nodes) represents the specific existence state of a trajectory or segment within a frame of the window. Each edge in the edge set (edge∈Edges) connects two different nodes belonging to the same physical target, indicating that these two states are related in time and space. To control the complexity of the graph and ensure real-time performance, candidate edges are only established between two nodes when the difference between their corresponding frame indices does not exceed 5 frames. Subsequently, every 10 frames processed, the system performs a global maximum weight matching algorithm on the local spatiotemporal graph. Successfully matched node pairs are considered as different segments of the same physical target and are then merged in chronological order. The identity ID is uniformly retained as the earlier created ID to ensure the long-term stability of the label.

7. The multi-target tracking method for autonomous driving scenarios according to claim 6, characterized in that, S5 specifically includes the following steps: to maintain the physical rationality of the output trajectory, perform pairwise collision detection on all confirmed trajectories within the current window; for any two trajectories, if there exists a frame such that their 3D bounding boxes intersect (comparison and union) ratio is... If an unacceptable penetration occurs, the intersection-union ratio (IUR) of the 3D bounding box is calculated using the following formula: ,in , These are the volumes of the two 3D bounding boxes. The volume of the overlapping area of ​​the two boxes; compare the confidence levels of the two trajectories in the current frame. and Select the one with lower confidence level as the adjustment target; calculate the overlap depth along the line connecting the centers of the two frames. The selected trajectory is translated outward along the line connecting the centers of the two frames. The distance; the outward translation The distance only affects the output trajectory and does not feed back to the internal state of the Kalman filter, thus avoiding causal violations; when multiple trajectories overlap, they are adjusted sequentially from low to high confidence; finally, all output trajectories satisfy the physical constraint of non-penetration, and the identity is effectively maintained after long-term occlusion.

8. A car, characterized in that, It includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement the multi-target tracking method for autonomous driving scenarios as described in any one of claims 1 to 7.

Images