A 3D multi-target tracking method, product, medium and system

Abstract

The present invention discloses a 3D multi-target tracking method, product, medium and system, the method comprising the steps of: S1, obtaining the original point cloud data of the vehicle and preprocessing it to obtain the preprocessed point cloud data; S2, extracting the features in the preprocessed point cloud data, obtaining the target detection result and generating a heat map; S3, performing motion compensation on the spatial position and orientation of the target by associating the heat map of the multi-frame point cloud; S4, performing confidence evaluation on the corrected target in the heat map, and adjusting the detection and tracking strategy according to the evaluation result. The present invention can significantly improve the accuracy and stability of tracking.

Description

Technical Field

[0001] The present invention mainly relates to the field of target tracking technology, and specifically to a 3D multi-target tracking method, product, medium and system. Background Art

[0002] With the rapid development of autonomous driving technology, 3D multi-object tracking (MOT) has become a key task in vehicle perception systems. MOT aims to accurately track multiple objects in real time in complex environments. Its applications span a wide range of fields, including autonomous driving, drone flight, and security surveillance. In autonomous driving scenarios, vehicles typically rely on multiple sensors, such as LiDAR, cameras, and millimeter-wave radar, to obtain detailed information about their surroundings. LiDAR, due to its ability to accurately acquire high-resolution 3D point cloud data and its strong environmental adaptability, has been widely used for object detection and tracking. However, existing 3D object tracking methods still face numerous challenges. First, in complex traffic environments, the appearance and motion characteristics of objects are diverse and interfere with each other, resulting in insufficient object tracking accuracy. Improving the ability to resolve different objects is an urgent issue. Second, autonomous driving places extremely high real-time demands on tracking systems, and existing methods often struggle to meet this requirement due to their high computational load and slow processing speed. Furthermore, occlusion between objects is common in real-world scenarios. Especially in dense traffic environments, mutual occlusion of multiple objects can lead to tracking failures. Furthermore, the motion of the vehicle itself and the target vehicle poses additional challenges to tracking. Traditional tracking methods are often not precise enough when processing motion compensation, resulting in large errors in the target position and direction. Existing 3D multi-target tracking methods mainly rely on point cloud data from lidar, and track targets through feature extraction, target detection, and trajectory management. Although these methods can perform well under ideal conditions, their tracking accuracy and stability are still insufficient in complex environments. In particular, for targets with different motion patterns, existing methods have limited ability to process motion compensation, which often leads to tracking errors in the target position. In addition, due to the lack of effective assessment of target confidence, existing methods perform poorly when dealing with complex scenarios such as occlusion and target disappearance. Therefore, improving the accuracy, real-time performance, and adaptability of 3D multi-target tracking to complex scenarios remains an important topic in current research on autonomous driving technology. Summary of the Invention

[0003] In response to the technical problems existing in the prior art, the present invention provides a 3D multi-target tracking method, product, medium and system for improving target tracking accuracy.

[0004] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0005] A 3D multi-target tracking method comprises the following steps:

[0006] S1. Obtaining the original point cloud data of the vehicle and preprocessing it to obtain preprocessed point cloud data;

[0007] S2. Extract features from the preprocessed point cloud data, obtain target detection results and generate a heat map;

[0008] S3, motion compensation is performed on the spatial position and orientation of the target by correlating the heatmaps of multiple frame point clouds;

[0009] S4. Perform confidence evaluation on the corrected target in the heat map and adjust the detection and tracking strategy based on the evaluation results.

[0010] Preferably, in step S3, according to the detection head speed Motion compensation is performed on the target spatial position in the Lidar coordinate system. Specifically, in the tracking result of the previous frame, the target spatial position is compensated using the following formula. The compensated position information is obtained as the matching object of the point cloud heat map of the current frame.

[0011]

[0012] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0013] Preferably, in step S3, the tracking speed in the Lidar coordinate system is Motion compensation is performed on the target's spatial position. Specifically, in the tracking result of the previous frame, the target's spatial position is compensated by the following formula, and the compensated position information is obtained as the matching object of the point cloud heat map of the current frame;

[0014]

[0015] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0016] Preferably, in step S3, according to the tracking speed in the World coordinate system Perform motion compensation on the spatial position of the target, specifically:

[0017] Transfer the target position in the Lidar coordinate system to the World coordinate system to obtain the target's velocity information position compensation in the World coordinate system:

[0018]

[0019] Where, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system; Represents the target coordinate value in the World coordinate system of the previous frame; Represents the time difference between two frames of point cloud;

[0020] After position compensation is completed, the target position information in the updated World coordinate system is returned to the Lidar coordinate system of the current frame.

[0021] Preferably, in step S3, the detection head speed is determined based on the confidence level in the Lidar coordinate system. and tracking speed Motion compensation of the target spatial position is performed by detecting the head velocity Tracking speed in Lidar coordinate system Combine to get the composite speed :

[0022]

[0023] Where Scores represents the detection head speed The confidence value of the Lidar coordinate system is given a higher confidence value, and the higher the confidence value, the higher the weight value of the detection head speed is. Assign a higher weight value;

[0024] The composite velocity in the Lidar coordinate system is applied to the displacement compensation using the following formula to obtain the updated position coordinates:

[0025]

[0026] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0027] Preferably, in step S3, the specific formula for performing motion compensation on the orientation of the target is:

[0028]

[0029] Where, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; Represents the time difference between two frames of point cloud; Indicates that the corresponding target is The angular velocity at the moment is calculated as:

[0030]

[0031] Where, Indicates the The orientation angle at the moment;

[0032] The rotation caused by the vehicle's motion is obtained by converting the Lidar coordinate system of the previous and next two frames to the World coordinate system. The specific calculation formula is:

[0033]

[0034] in, 、 Represents the rotation matrix from the previous moment Lidar coordinate system to the current frame Lidar coordinate system The element value of is calculated by calculating the arc tangent of these two elements to obtain the corresponding angle information, that is, the angle deviation value caused by the vehicle;

[0035] in The calculation is obtained by the formula:

[0036]

[0037] Where, and They represent the inverse of the rotation matrix from the current frame's Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame's Lidar coordinate system to the World coordinate system, respectively.

[0038] Preferably, in step S4, the specific process of adjusting the detection and tracking strategy according to the evaluation result is:

[0039] First, we extract the features of the current frame to get T The heat map information with confidence higher than the threshold M is retained;

[0040] For the input T Frame point cloud feature extraction to obtain T The heat map information is used to compensate the obstacles to the current frame T through motion compensation. Lidar coordinate system with T The heat map of the frame above the threshold M is matched; in the matching process, the corresponding position confidence of the heat map of the previous and next two frames is processed to obtain the weighted heat map information; the updated heat map information is divided into the current frame T according to the confidence With the previous frame T Registered heat map information, unregistered T Heatmap information and unregistered T Frame heatmap information;

[0041] The current frame T With the previous frame T The registered heat map information indicates that the current frame is successfully matched with the historical trajectory to obtain ID information;

[0042] Unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, that is, unmatched track information; unmatched track information will not be actively deleted, but long-term unmatched track information will be eliminated according to the characteristics of matching weight half-decay;

[0043] Unregistered T Frame heat map information represents the last frame T There is a lack of track heatmap that can match the current frame, that is, the new target information; a confidence level N is set for the establishment of the new target track. It is believed that the target with a higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for the unregistered T above the threshold N, A new track is created directly from the frame heat map. For heat map information below the threshold N, it is considered that no registration with the history is obtained and the confidence score is low. It is considered to be a distant target with incomplete features or a model misdetection, and is selected as the heat map information that needs to be discarded.

[0044] Preferably, in step S2, the features in the point cloud data include geometric features, motion features, depth features and spatial features.

[0045] Preferably, in step S1, the preprocessing includes denoising, filtering and data enhancement.

[0046] The present invention also discloses a computer program product, comprising a computer program, which executes the steps of the above method when executed by a processor.

[0047] The present invention further discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described above are executed.

[0048] The present invention also discloses a 3D multi-target tracking system, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, and when the computer program is run by the processor, the steps of the above method are executed.

[0049] Compared with the prior art, the advantages of the present invention are:

[0050] The present invention begins with the preprocessing of point cloud data, extracts features and generates a heatmap through a deep learning model, then ensures the stability of the system in dynamic environments through efficient multi-target tracking and innovative position and orientation compensation mechanisms. Finally, through confidence hierarchical association and trajectory management, the accuracy and robustness of multi-target tracking are further improved.

[0051] The present invention deeply integrates the principle of spatiotemporal interactive joint detection, and realizes the refined processing and feature extraction of raw point cloud data through a carefully designed data processing flow. It not only includes an efficient feature extraction mechanism that can accurately capture the unique identification information of the target, but also incorporates motion compensation technology to cope with the challenges brought by the movement of the ego vehicle and the target vehicle. In addition, it also adopts a confidence hierarchical association strategy, which significantly improves the tracking accuracy and stability by performing multi-level evaluation and association of the confidence of the target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 FIG. 4 is a flow chart of a 3D multi-target tracking method according to an embodiment of the present invention.

[0053] Figure 2 This is a flow chart of the 3D multi-target tracking method of the present invention during specific application.

[0054] Figure 3 Schematic diagram of the misalignment of target information of two frames of point clouds at different times in the Lidar coordinate system in the present invention.

[0055] Figure 4 Schematic diagram of target speed in the Lidar coordinate system of the present invention.

[0056] Figure 5 Schematic diagram of target position deviation in the World coordinate system of the present invention.

[0057] Figure 6 Schematic diagram of Lidar coordinate position compensation for detection head velocity in the present invention.

[0058] Figure 7 Schematic diagram of the Lidar coordinate target orientation deviation in the present invention.

[0059] Figure 8 This is an example diagram of the Heatmap confidence mean calculation in the present invention.

[0060] Figure 9 Schematic diagram of the multi-level processing of heat maps in the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0062] like Figure 1 As shown, the 3D multi-target tracking method according to an embodiment of the present invention includes the following steps:

[0063] S1. Data preprocessing:

[0064] First, the input raw vehicle point cloud data is preprocessed, including denoising and filtering operations, to eliminate interference factors, improve data quality, and ensure that the point cloud data used in subsequent processing is cleaner and more accurate. In addition, data enhancement techniques such as random rotation, translation, scale transformation, and random point discarding are also used to further enhance the expressiveness of the point cloud.

[0065] S2. Deep learning feature extraction and heatmap generation:

[0066] First, a deep learning model performs in-depth analysis and feature extraction on the preprocessed point cloud data. This model, through a multi-layer neural network architecture, gradually extracts key features from each frame of point cloud data. These key features reflect the target's geometry, motion state, and spatial distribution. After feature extraction, the system generates a corresponding heatmap to visually display the target's position distribution and feature density in 3D space. The heatmap not only captures the distribution pattern of different targets in the point cloud data, but also contains rich target-related information, such as the target's size, shape, and dynamic changes in space.

[0067] Key features refer to the multidimensional information extracted from each frame of point cloud data through a deep learning model, including the point cloud distribution pattern in space and key data related to the target. The heatmap generated by processing these features not only displays the spatial arrangement of the point cloud but also captures important information such as the target's position and shape, becoming an indispensable core foundation for the multi-target tracking process. Specifically, these key features cover the following aspects: first, geometric features. The geometric shape and distribution pattern in the point cloud can help the system identify the outline and relative position of the object; second, motion features. By analyzing multi-frame point cloud data, the system can capture the object's motion trajectory and dynamic changes; and finally, depth and spatial features. The depth information and relative position of the point cloud are key to forming a 3D structure, helping to distinguish the front-to-back relationship of an object and its precise positioning in space. Through deep learning mining, these features are converted into important basic data for the target tracking system, supporting the system to achieve efficient and accurate multi-target tracking.

[0068] This process lays a solid foundation for subsequent multi-target tracking, ensuring that the system can accurately perceive and associate multiple targets, and improving the accuracy and stability of target detection.

[0069] S3. Construction of multi-target tracking module:

[0070] By combining heatmaps extracted through deep learning features, the system builds an efficient multi-target tracking module. This module seamlessly associates heatmaps from multiple point clouds in the LiDAR coordinate system, enabling precise joint detection and tracking of multiple targets.

[0071] To enhance its time series processing capabilities, the module introduces an iterative mechanism that combines the current frame's real-time information with accumulated data from past frames, simulating a memory effect similar to that of an LSTM (Long Short Term Memory) model. This mechanism ensures the system can effectively capture the dynamic changes of the target at different moments, particularly in complex dynamic environments, better adapting to the target's trajectory changes and the real-time requirements of the scene. This mechanism aims to achieve dynamic tracking of the target by combining data from past frames with information from the current frame. The LSTM model is characterized by its ability to handle long-term dependencies in time series tasks, retaining historical information through memory cells and dynamically adjusting its state. Although this method does not directly utilize the LSTM model, it simulates a similar memory effect by combining historical and current data through an iterative mechanism. This mechanism not only preserves the target's historical trajectory but also utilizes the characteristics of the current frame for adjustment, thereby improving the system's adaptability to dynamic changes. During the tracking process, this method continuously integrates information from multiple frames, enabling the system to effectively handle complex situations such as target occlusion and movement, maintaining tracking continuity and accuracy. This interaction and accumulation of spatiotemporal information, similar to the memory and update process in LSTM, enables the tracking system to exhibit greater stability and robustness in dynamic environments. This approach enables the system to accurately track multiple targets, maintaining tracking continuity and robustness, and minimizing tracking interruptions or deviations caused by target motion or occlusion.

[0072] Specifically, existing multi-target tracking is mostly performed using the TBD (Tracking By Detection) paradigm. The detection module, acting as an upstream task of the tracking module, outputs target information. The multi-target tracking module then assigns track attributes to this detected target information through methods such as matching and association. This approach results in a one-way relationship between detection and multi-target tracking, preventing the detection module from effectively utilizing the information gained from multi-target tracking to improve target detection accuracy.

[0073] To address this issue, the present invention constructs a mathematical model for joint detection multi-target tracking, analyzing its input and output information. It also describes a deep learning-based network model for joint detection 3D multi-target tracking, analyzing the network model structure and key points. Specifically, using LiDAR as a carrier, the present invention selects joint detection 3D multi-target tracking as the research object. To clarify the objectives of this task, as well as the content of the raw data and final results, a mathematical model for joint detection multi-target tracking is constructed, and the main data flows of the present invention are explained through this mathematical model.

[0074] The present invention uses continuous laser radar point cloud data as input information and ultimately outputs target recognition and target tracking of the laser radar point cloud data. The present invention simultaneously realizes detection and tracking tasks through a single model.

[0075] Record the input information of continuous laser point cloud , where P represents the set of laser point clouds in the time domain in the current scene; Represents at the moment Corresponding Laser point cloud data, each laser point cloud data consists of composition, represents the position of the laser point cloud, Represents intensity information.

[0076] The output information of joint detection and tracking is recorded as , where O represents the detection and tracking results of continuous input point cloud data; Represents at the moment Corresponding The detection and tracking information of each target is mainly composed of etc., among which Represents the spatial position and size information of the target, Represents the target's orientation angle and velocity information in the corresponding coordinate system. Represent the target’s category information, tracking ID and confidence information respectively. Goal Represents at different times The following belong to a real external obstacle.

[0077] Compared to joint detection methods that rely on pixel data as input, multi-target tracking methods that use point cloud data acquired from LiDAR scans require consideration of how to improve the model's feature extraction capabilities and how to better utilize the extracted information as a basis for combined target detection and multi-target tracking. The model architecture uses continuous multi-frame point clouds as input. It then extracts features from the raw point cloud, generates a heat map of the corresponding moment, and compensates for heat map motion in the temporal domain, ultimately outputting target detection and tracking results.

[0078] For example, Figure 2As shown, the first frame of point cloud data is input at time T1, and the point cloud data first obtains higher-dimensional point cloud features through feature extraction. The higher-dimensional point cloud features obtain information such as the target center point according to the decoder, and the target detection result of the first frame point cloud is obtained based on the center point and other information. The track information in the current scene is initialized according to the target information of the first frame; when the point cloud at time T1 completes target detection and multi-target tracking, the corresponding tracking results are stored, and the tracking results are motion compensated in the form of a heat map, and matched with the heat map generated by the features obtained from the input T2 frame point cloud, and the detection and multi-target tracking results are obtained by matching; the above operations on the point clouds of frames T1 and T2 are repeated to obtain the target detection and multi-target tracking results of the point clouds at other subsequent times.

[0079] Motion compensation mechanism:

[0080] The motion compensation mechanism is designed to solve the problem of Heatmap misalignment caused by the movement of the ego vehicle or target vehicle, ensuring the continuity and accuracy of multi-target tracking. This mechanism accurately calculates the target's velocity information in the Lidar coordinate system, combined with the uniform motion model, to compensate the center point of the Heatmap in real time. When the vehicle or target is in motion, the system predicts the target's position at the next moment based on the target's current speed and direction of movement, and corrects the displacement deviation caused by the movement. In this way, the misalignment of the target between different frames is effectively corrected, the tracking trajectory is kept continuous, and the accumulation of errors caused by motion is avoided. Through this motion compensation mechanism, the system can accurately track targets in high-speed or complex dynamic environments, significantly improving the accuracy and stability of multi-target tracking.

[0081] Specifically, when the ego vehicle and the target vehicle are in motion, the target position in the Lidar coordinate system may be offset, resulting in heatmap misalignment and affecting tracking accuracy. In order to solve this problem, the present invention accurately calculates the speed information of the target in the Lidar coordinate system and combines it with the uniform motion model to compensate the center point of the heatmap. This means that in each frame, the system will predict and correct the spatial position of the target based on the speed and direction of movement of the target to eliminate the displacement deviation caused by movement. Through this compensation mechanism, the tracking system can not only maintain the continuity of the target, but also improve the accuracy of tracking in a dynamic environment, ensuring accurate multi-target tracking even in high-speed or complex motion scenes, effectively eliminating the displacement deviation caused by movement, and ensuring the continuity and accuracy of tracking.

[0082] This module takes the heatmap corresponding to the target tracking in the previous frame as input information. Due to the motion of the vehicle and other vehicles, motion compensation is performed before the tracking detection heatmap matching process. The target's speed information is obtained through the CV (Computer Vision) model to correct the target's heatmap center point and orientation in preparation for subsequent matching.

[0083] Based on the motion compensation of the target in the current Lidar coordinate system, during the motion process, due to the motion of the ego vehicle and the target vehicle, the target vehicle is often in different positions in the ego vehicle Lidar coordinate system at different times. However, this paper associates the historical point cloud heat map of the previous and next frames based on the current Lidar coordinate system, which is strongly dependent on the position information of the target in the Lidar coordinate system. Therefore, it is very important to correct and compensate the position of the target in the Lidar. Figure 3 As shown in the figure, the target information of the two frames of point clouds at different times has been greatly misaligned in the Lidar coordinate system.

[0084] In the target LiDAR misalignment diagram, the left and right frames represent the point clouds for consecutive frames Tn−1 and Tn, respectively. Circles of the same color in the left and right images represent the same object. For example, the green special vehicle and the car in the red circle in the right image have both shifted significantly downward relative to their positions at the previous moment Tn−1 in the LiDAR coordinate system. The vehicle represented by the blue circle in the right image has not changed much relative to its position at the previous moment Tn−1 in the LiDAR coordinate system. Analysis shows that due to the movement of the ego vehicle's LiDAR and the movement of the external target vehicle, there is a non-fixed offset in the coordinates of the point clouds between the two consecutive frames.

[0085] In the present invention, the displacement deviation between the previous and next frames is eliminated mainly through a uniform motion model.

[0086] Spatial position and orientation compensation:

[0087] The spatial position and orientation compensation mechanism aims to solve the problem of the position difference between the target in the Lidar coordinate system and the World coordinate system, as well as the change of the target orientation angle with movement.

[0088] First, spatial position compensation uses the velocity information of the detection head and the velocity data calculated from the tracking results to accurately adjust the position of the target in the two coordinate systems to ensure that the target position in different coordinate systems remains consistent.

[0089] Secondly, orientation compensation dynamically corrects the orientation information in the Heatmap based on possible orientation changes of the target during movement.

[0090] This compensation mechanism enables the system to accurately track targets in complex dynamic environments, ensuring not only precise spatial positioning but also the correct orientation of the targets, further improving the accuracy and stability of multi-target tracking. This dual compensation mechanism effectively addresses coordinate misalignment and orientation deviations caused by vehicle motion, making tracking results more reliable.

[0091] Specifically, velocity information is often difficult to directly detect in the detection module. However, in the field of multi-target tracking, velocity information can be easily calculated by analyzing the position information of the same target at different time points. This calculated velocity information can better explain the current target's motion pattern within the previous time range and its motion trend within a certain period of time in the future, thereby achieving more accurate target prediction information and improving the accuracy and robustness of target perception.

[0092] like Figure 4 As shown in the figure, the red arrow represents the lidar's X-axis, the green arrow represents the lidar's Y-axis, and O represents the lidar coordinate system origin. The blue and purple arrows represent the target's velocity in the world coordinate system and the lidar's velocity in the world coordinate system, respectively. The velocity in the world coordinate system only represents the target's own velocity. This velocity is calculated by projecting the target into the reference world coordinate system and calculating the position deviation.

[0093] like Figure 5 The figure shows the target position deviation in the World coordinate system.

[0094] The position deviation and speed calculation formula of the tracking result target are:

[0095]

[0096]

[0097] in, Indicates the target is The offset between two frames on the axis, Indicates the target is The offset between two frames on an axis; Indicates the target is The offset between the two frames on the axis; V represents the actual speed information of the final target in the reference world coordinate system, Indicates the time difference between two frames of point cloud that produces position offset.

[0098] In step S3, target position compensation is performed using the detection head speed and tracking speed in the Lidar coordinate system, as well as target position compensation in the World coordinate system. , tracking speed in Lidar coordinate system And the tracking speed in the World coordinate system Through these three types of speed information, the target spatial position compensation of the point cloud heat map is achieved through different speeds and their combinations in the time domain. Specifically:

[0099] 1) Detection head speed Compensate the target position in the Lidar coordinate system: In the subtask of the detection head, output the speed information in two directions. These two speeds are respectively 、 In the tracking result of the previous frame, the target space position is compensated by the following formula, and the compensated position information is obtained as the matching object of the point cloud heat map of the current frame;

[0100]

[0101] in, Represents the updated position information of the previous frame after motion compensation in the Lidar coordinate system; Indicates the location information of the target's detection and tracking results obtained in the previous frame; 、 They represent the detection head velocity information and the time difference between two frames of point cloud respectively.

[0102] 2) If Figure 6 Shown is the tracking speed in the Lidar coordinate system Compensation for target position: The target position compensation based on the tracking speed in the Lidar coordinate system is similar to the target position compensation based on the speed output by the detection head in the Lidar coordinate system. The speed information obtained from the tracking result in the Lidar coordinate system is used to compensate the detection and tracking result heat map of the previous frame:

[0103]

[0104] in, Represents the target's motion speed in the Lidar coordinate system obtained by calculating the detection and tracking results of the previous frame.

[0105] 3) Tracking speed in the World coordinate system Compensate the target position: To compensate the target position in the World coordinate system, you first need to transfer the target in the Lidar coordinate system to the World coordinate system. The transfer method is shown in the following formula:

[0106]

[0107] in Represents the coordinate value of the previous frame under the World, where the coordinate value is as follows:

[0108]

[0109] Among them, x, y, and z represent the position of the laser point cloud.

[0110] As the first The rotation and translation matrix of the frame represents the rotation and translation matrix of the current frame point cloud from the Lidar coordinate system to the World coordinate system, and its size is The specific expression of the matrix is:

[0111]

[0112] in, and Represent the corresponding rotation matrix and translation matrix respectively, where is a size of The matrix, is a size of Matrix; 0 in the above formula represents a 1 The zero vector of , 1 represents 1 The unit array.

[0113] The target position and other information in the Lidar coordinate system obtained by feature extraction are calculated by the above formula to obtain the spatial position expressed in the World coordinate system. The speed information position compensation of the target obtained from the tracking result in the World coordinate system is calculated by the above formula:

[0114]

[0115] Where, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system.

[0116] After completing the position compensation, the target position information in the updated World coordinate system needs to be returned to the coordinate system of the current frame, as shown in the following formula:

[0117]

[0118] Where, Represents the inverse of the rotation and translation matrix of the current frame point cloud from the Lidar coordinate system to the World coordinate system. This is used to obtain the new position information of the Lidar target in the Lidar coordinate system after the velocity information position compensation in the World coordinate system, which serves as the input for the subsequent matching with the current frame.

[0119] 4) Compensation of target position based on confidence-based detection head speed and tracking speed in the Lidar coordinate system: The first three position compensations are all achieved through a single speed information. This method has high requirements for single speed information and requires it to have extremely high accuracy. However, since the speed calculation of the current frame is usually performed based on the displacement deviation between the current frame and the previous frame, the calculated speed has a certain lag. At the same time, since the speed information of the detection head is predicted through deep learning, the obtained speed information is also positively correlated with the confidence of the target. Therefore, the predicted speed of the detection head is combined with the speed obtained by tracking in the Lidar coordinate system through the following formula:

[0120]

[0121] Where Scores represents the speed predicted by the detection head The confidence value of the detection head is given a higher weight. A higher confidence value gives a higher weight to the detection head's predicted velocity. A lower confidence value indicates that the detection head's predicted velocity is less reliable to a certain extent, which may be caused by reasons such as occlusion. Therefore, a higher weight is given to the velocity in the Lidar coordinate system obtained through tracking. Finally, the composite velocity in the Lidar coordinate system is applied to the displacement compensation through the following formula to obtain the updated position coordinates:

[0122] .

[0123] While we've already discussed compensating for the target's spatial position, the target's orientation is also crucial for perception. Between two point cloud frames, the target's orientation in LiDAR coordinates may change due to the motion of the vehicle or other vehicles. Therefore, after compensating for the target's position, it's also crucial to correct the target's orientation.

[0124] like Figure 7 As shown in the figure, in the point cloud coordinate system at two moments, the target's orientation angle has a certain deviation due to its own steering and the movement of the vehicle.

[0125] Through the above analysis, target orientation compensation is mainly divided into two parts: the orientation angle change caused by the vehicle's motion and the angular velocity information of the other vehicle. The specific calculation formula is as follows:

[0126]

[0127] Where, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; It represents the angular velocity of the corresponding target, which is calculated as follows:

[0128]

[0129] In the formula, due to the The angular velocity of the target at time t, so Indicates the The orientation angle at the moment;

[0130] The rotation caused by the vehicle's motion is obtained by converting the Lidar coordinate system of the previous and next two frames to the World coordinate system. The specific calculation formula is:

[0131]

[0132] in, 、 Represents the rotation matrix from the previous moment Lidar coordinate to the current frame World coordinate The element value of , by calculating the arc tangent of these two elements to obtain the corresponding angle information, the angle deviation value caused by the vehicle can be obtained. The calculation is obtained by the formula:

[0133]

[0134] Where, and They represent the inverse of the rotation matrix from the current frame's Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame's Lidar coordinate system to the World coordinate system, respectively.

[0135] S4. Confidence hierarchical multi-level association strategy:

[0136] The confidence hierarchical multi-level association strategy aims to improve the accuracy and robustness of target tracking, especially to effectively reduce the risk of mistracking and missed tracking in complex scenarios.

[0137] like Figure 8 Shown is an example of calculating the Heatmap confidence mean.

[0138] This strategy first evaluates the confidence of the targets in the Heatmap, carefully layers targets of different confidence levels, and combines the high- and low-scoring target association strategy of the Bytetrack algorithm. For targets with higher confidence levels, a direct association method is used for tracking; for targets with lower confidence levels, a more sophisticated multi-level association strategy is used to ensure that the system can accurately match and track targets even at low confidence levels, effectively reducing the risk of mistracking and missed tracking, and significantly improving the performance of the entire tracking system.

[0139] This strategy combines the Bytetrack algorithm's high- and low-score target association mechanism to further enhance the system's tracking capabilities in situations with occlusion, overlapping targets, or complex environments. This layered approach allows the system to dynamically adjust the association method, making the entire tracking process more precise and stable, significantly improving tracking performance, particularly in multi-target scenarios.

[0140] Specifically, Bytetrack assumes that detection boxes below a set threshold will not be used for ID matching. This results in boxes with low confidence due to occlusion or sparse target point clouds being simply discarded, leading to large areas of missed detection and trajectory fragmentation. Therefore, Bytetrack proposes a simple and efficient data association method. The idea is to perform multi-stage association on high- and low-scoring objects, removing some low-scoring objects for more efficient performance.

[0141] like Figure 9 As shown, the multi-level association method based on confidence level in the present invention is specifically as follows:

[0142] First, we extract the features of the current frame to get T The heat map information with confidence higher than M is retained;

[0143] For the input T The heat map information is obtained by extracting the frame point cloud features, and the obstacles are compensated to the current frame T through motion compensation. Lidar coordinate system with T The frame with a heat map higher than the threshold M is matched. During the matching process, the corresponding position confidence of the heat maps of the previous and next two frames is processed to obtain weighted heat map information. The updated heat map information is divided into the current frame T according to the confidence With the previous frame T Registered heat map information, unregistered T Heatmap information and unregistered T Frame heatmap information.

[0144] The current frame T With the previous frame T The registered heat map information indicates that the current frame successfully matches the historical trajectory to obtain ID information; the unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, and the track information is not matched; the unmatched track information will not be deleted actively, but the long-term unmatched track information will be eliminated according to the half-decay characteristics of the matching weight. Frame heat map information represents the last frame T There is a lack of track heatmap that can match the current frame, that is, the new target information. A higher confidence level N is set for the establishment of the new target track. It is believed that the target with higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for the current frame T higher than the threshold N, For heatmap information that is not registered above the threshold N, a new track is created directly. For heatmap information below the threshold N, it is considered that no registration with the history is obtained and the confidence score is low. It may be a distant target with incomplete features or a model misdetection, and is selected as the heatmap information that needs to be discarded.

[0145] S5. Result output and tracking management:

[0146] Tracking data is continuously updated and adjusted through confidence and multi-target association, ultimately outputting the trajectory and status of each target. The system manages point cloud data and target trajectories at every moment, dynamically adjusting the target's status to ensure continuous and accurate tracking.

[0147] The present invention begins with the preprocessing of point cloud data, extracts features and generates a heatmap through a deep learning model, then ensures the stability of the system in dynamic environments through efficient multi-target tracking and innovative position and orientation compensation mechanisms. Finally, through confidence hierarchical association and trajectory management, the accuracy and robustness of multi-target tracking are further improved.

[0148] The present invention deeply integrates the principle of spatiotemporal interactive joint detection, and realizes the refined processing and feature extraction of raw point cloud data through a carefully designed data processing flow. It not only includes an efficient feature extraction mechanism that can accurately capture the unique identification information of the target, but also incorporates motion compensation technology to cope with the challenges brought by the movement of the ego vehicle and the target vehicle. In addition, it also adopts a confidence hierarchical association strategy, which significantly improves the tracking accuracy and stability by performing multi-level evaluation and association of the confidence of the target tracking results.

[0149] An embodiment of the present invention further discloses a computer program product, including a computer program. When executed by a processor, the computer program performs the steps of the above method. An embodiment of the present invention further discloses a computer-readable storage medium, on which the computer program is stored. When executed by a processor, the computer program performs the steps of the above method. An embodiment of the present invention further discloses a computer device, including a memory and a processor connected to each other, the memory storing a computer program. When executed by the processor, the computer program performs the steps of the above method.

[0150] The present invention can implement all or part of the process steps in the above-described method embodiments through hardware associated with computer program instructions. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. Computer-readable storage media include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory and accessing data stored in the memory. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0151] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A 3D multi-target tracking method, characterized in that: Includes steps: S1. Acquire original point cloud data of the vehicle and perform preprocessing to obtain preprocessed point cloud data; S2, extracting features from the preprocessed point cloud data, obtaining target detection results and generating a heat map; S3, motion compensation is performed on the spatial position and orientation of the target by associating the heat map of multiple frame point clouds; S4, conduct confidence evaluation on the corrected target in the thermal map, and adjust the detection and tracking strategy according to the evaluation result; In step S3, the specific formula for motion compensation of the target orientation is: In the formula, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; Represents the time difference between two frames of point cloud; Indicates that the corresponding target is Angular velocity at the moment; Indicates the rotation angle deviation value caused by the vehicle's motion.

2. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, according to the detection head speed Motion compensation is performed on the target space position in the Lidar coordinate system. Specifically, in the tracking result of the previous frame, the target space position is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame; in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

3. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, the tracking speed in the Lidar coordinate system is Motion compensation is performed on the spatial position of the target. Specifically, in the tracking result of the previous frame, the spatial position of the target is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame; in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

4. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, according to the tracking speed in the World coordinate system Motion compensation is performed on the spatial position of the target, specifically: Transfer the target position in the Lidar coordinate system to the World coordinate system to obtain the speed information position compensation of the target in the World: In the formula, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system; Represents the target coordinate value in the World coordinate system of the previous frame; Represents the time difference between two frames of point cloud; After completing the position compensation, the target position information in the updated World coordinate system is returned to the Lidar coordinate system of the current frame.

5. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, the detection head velocity based on the confidence in the Lidar coordinate system is Tracking speed Motion compensation for the target spatial position is performed by detecting the head speed Tracking speed in Lidar coordinate system Combine to get the composite speed : In the formula, Scores represents the detection head speed The confidence value of the Lidar coordinate system is given a higher weight for the detection head speed, and a lower confidence value for the tracking speed in the Lidar coordinate system. Assign a higher weight value; The composite velocity in the Lidar coordinate system is applied to the displacement compensation through the following formula to obtain the updated position coordinates: in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

6. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: The calculation formula is: In the formula, Indicates The orientation angle at the moment; Represents the rotation caused by the vehicle's motion, which is obtained by the rotation matrix of the Lidar coordinate system to the World coordinate system of the previous and next two frames. The specific calculation formula is: in, , Represents the rotation matrix from the previous moment's Lidar coordinate system to the current frame's Lidar coordinate system The element value of , the arc tangent is calculated through these two elements to obtain the corresponding angle information, that is, the angle deviation value caused by the vehicle; in The calculation is obtained by the formula: In the formula, and They represent the inverse of the rotation matrix from the current frame Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame Lidar coordinate system to the World coordinate system respectively.

7. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S4, the specific process of adjusting the detection and tracking strategy according to the evaluation results is as follows: First, we extract the features of the current frame to get T The heat map information with confidence higher than the threshold M is retained; For the input T Frame point cloud feature extraction to obtain T The heat map information is used to compensate the obstacles to the current frame T through motion compensation. The Lidar coordinate system is the same as T The heat map of the frame above the threshold M is matched; in the matching process, the corresponding position confidence of the heat map of the previous and next two frames is processed to obtain the weighted heat map information; the updated heat map information is divided into the current frame T according to the confidence. T The registered heat map information and the unregistered T Heatmap information and unregistered T Frame heat map information; The current frame T T The registered heat map information indicates that the current frame successfully matches the historical trajectory to obtain ID information; Unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, that is, unmatched track information; the unmatched track information will not be deleted actively, but the long-term unmatched track information will be eliminated according to the characteristics of the half-decay of the matching weight; Unregistered T The frame heat map information represents the last frame T There is a lack of track heatmaps that can match the current frame, that is, new target information; a confidence level N is set for the establishment of new target tracks, and it is believed that the target that can obtain a higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for unregistered T above the threshold N, The frame heat map directly creates a new track. For heat map information below the threshold N, it is considered that the registration with the history is not obtained and the confidence score is low. It is a distant target with incomplete features or a model misdetection, and is selected as the heat map information that needs to be discarded.

8. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S2, the features in the point cloud data include geometric features, motion features, depth features and spatial features.

9. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S1, the preprocessing includes denoising, filtering and data enhancement.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are performed.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 9.

12. A 3D multi-target tracking system, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 9.

Images