3D multi-target tracking method, product, medium and system

Abstract

The invention discloses a 3D multi-target tracking method, a 3D multi-target tracking product, a 3D multi-target tracking medium and a 3D multi-target tracking system, and the method comprises the steps: S1, obtaining original point cloud data of a vehicle, and carrying out the preprocessing of the original point cloud data, and obtaining the preprocessed point cloud data; s2, extracting features in the preprocessed point cloud data to obtain a target detection result and generate a thermodynamic diagram; s3, performing motion compensation on the spatial position and orientation of the target by associating the thermodynamic diagrams of the multiple frames of point clouds; and S4, carrying out confidence evaluation on the corrected target in the thermodynamic diagram, and adjusting a detection tracking strategy according to an evaluation result. According to the invention, the tracking precision and stability can be obviously improved.

Description

Technical Field

[0001] The present invention mainly relates to the field of target tracking technology, and specifically relates to a 3D multi-target tracking method, product, medium and system. Background Art

[0002] With the rapid development of autonomous driving technology, 3D multi-object tracking (MOT) has become one of the key tasks in vehicle perception systems. Multi-object tracking aims to track multiple objects in real time and accurately in complex environments. Its applications cover a wide range of fields, including autonomous driving, drone flight, and security monitoring. In autonomous driving scenarios, vehicles usually rely on a variety of sensors, such as Lidar, cameras, and millimeter-wave radars, to obtain detailed information about the surrounding environment. Among them, Lidar has been widely used in target detection and tracking because it can accurately obtain high-resolution 3D point cloud data and has strong environmental adaptability. However, existing 3D target tracking methods still face many challenges. First, in a complex traffic environment, the appearance and motion characteristics of the target are diverse and interfere with each other, resulting in insufficient target tracking accuracy. Improving the ability to distinguish between different targets has become an urgent problem to be solved. Second, autonomous driving has extremely high real-time requirements for the tracking system, and existing methods are often difficult to meet this requirement due to large computational complexity and slow processing speed. In addition, there is often occlusion between targets in actual scenarios. Especially in dense traffic environments, the mutual occlusion of multiple targets will cause tracking failure. Furthermore, the movement of the vehicle itself and the target vehicle brings additional challenges to tracking. Traditional tracking methods are often not accurate enough when dealing with motion compensation, resulting in large errors in the target position and direction. Existing 3D multi-target tracking methods mainly rely on the point cloud data of LiDAR to track targets through feature extraction, target detection and trajectory management. Although these methods can perform tasks well under ideal conditions, their tracking accuracy and stability are still insufficient in complex environments. In particular, for targets with different motion modes, the existing methods have limited ability to deal with motion compensation, which often leads to tracking errors in the target position. In addition, due to the lack of effective evaluation of target confidence, existing methods perform poorly when dealing with complex scenes such as occlusion and target disappearance. Therefore, improving the accuracy, real-time performance and adaptability to complex scenes of 3D multi-target tracking remains an important topic in the current research of autonomous driving technology. Summary of the invention

[0003] In view of the technical problems existing in the prior art, the present invention provides a 3D multi-target tracking method, product, medium and system for improving target tracking accuracy.

[0004] In order to solve the above technical problems, the technical solution proposed by the present invention is: A 3D multi-target tracking method comprises the following steps: S1. Acquire original point cloud data of the vehicle and perform preprocessing to obtain preprocessed point cloud data; S2, extracting features from the preprocessed point cloud data, obtaining target detection results and generating a heat map; S3, motion compensation is performed on the spatial position and orientation of the target by associating the heat map of multiple frame point clouds; S4. Perform confidence evaluation on the corrected target in the heat map and adjust the detection and tracking strategy based on the evaluation results.

[0005] Preferably, in step S3, according to the detection head speed Motion compensation is performed on the target space position in the Lidar coordinate system. Specifically, in the tracking result of the previous frame, the target space position is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame;

[0006] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0007] Preferably, in step S3, the tracking speed in the Lidar coordinate system is Motion compensation is performed on the spatial position of the target. Specifically, in the tracking result of the previous frame, the spatial position of the target is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame;

[0008] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0009] Preferably, in step S3, according to the tracking speed in the World coordinate system Motion compensation is performed on the spatial position of the target, specifically: Transfer the target position in the Lidar coordinate system to the World coordinate system to obtain the speed information position compensation of the target in the World:

[0010] In the formula, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system; Represents the target coordinate value in the World coordinate system of the previous frame; Represents the time difference between two frames of point cloud; After completing the position compensation, the target position information in the updated World coordinate system is returned to the Lidar coordinate system of the current frame.

[0011] Preferably, in step S3, the detection head speed based on the confidence in the Lidar coordinate system is Tracking speed Motion compensation for the target spatial position is performed by detecting the head speed Tracking speed in Lidar coordinate system Combine to get the composite speed :

[0012] In the formula, Scores represents the detection head speed The confidence value of the Lidar coordinate system is given a higher weight for the detection head speed, and a lower confidence value for the tracking speed in the Lidar coordinate system. Assign a higher weight value; The composite velocity in the Lidar coordinate system is applied to the displacement compensation through the following formula to obtain the updated position coordinates:

[0013] in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

[0014] Preferably, in step S3, the specific formula for motion compensation of the target orientation is:

[0015] In the formula, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; Represents the time difference between two frames of point cloud; Indicates that the corresponding target is The angular velocity at the moment is calculated as:

[0016] In the formula, Indicates The orientation angle at the moment; Represents the rotation caused by the vehicle's motion, which is obtained by the rotation matrix of the Lidar coordinate system to the World coordinate system of the previous and next two frames. The specific calculation formula is:

[0017] in, , Represents the rotation matrix from the previous moment's Lidar coordinate system to the current frame's Lidar coordinate system The element value of , the arc tangent is calculated through these two elements to obtain the corresponding angle information, that is, the angle deviation value caused by the vehicle; in The calculation is obtained by the formula:

[0018] In the formula, and They represent the inverse of the rotation matrix from the current frame Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame Lidar coordinate system to the World coordinate system respectively.

[0019] Preferably, in step S4, the specific process of adjusting the detection and tracking strategy according to the evaluation result is: First, we extract the features of the current frame to get T The heat map information with confidence higher than the threshold M is retained; For the input T Frame point cloud feature extraction to obtain T The heat map information is used to compensate the obstacles to the current frame T through motion compensation. The Lidar coordinate system is the same as T The heat map of the frame above the threshold M is matched; in the matching process, the corresponding position confidence of the heat map of the previous and next two frames is processed to obtain the weighted heat map information; the updated heat map information is divided into the current frame T according to the confidence. T The registered heat map information and the unregistered T Heatmap information and unregistered T Frame heatmap information; The current frame T T The registered heat map information indicates that the current frame successfully matches the historical trajectory to obtain ID information; Unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, that is, unmatched track information; the unmatched track information will not be deleted actively, but the long-term unmatched track information will be eliminated according to the characteristics of the half-decay of the matching weight; Unregistered T The frame heat map information represents the last frame T There is a lack of track heatmaps that can match the current frame, that is, new target information; a confidence level N is set for the establishment of new target tracks, and it is believed that the target that can obtain a higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for unregistered T above the threshold N, The frame heat map directly creates a new track. For heat map information below the threshold N, it is considered that the registration with the history is not obtained and the confidence score is low. It is a distant target with incomplete features or a model misdetection, and is selected as the heat map information that needs to be discarded.

[0020] Preferably, in step S2, the features in the point cloud data include geometric features, motion features, depth features and spatial features.

[0021] Preferably, in step S1, the preprocessing includes denoising, filtering and data enhancement.

[0022] The present invention also discloses a computer program product, comprising a computer program, wherein the computer program executes the steps of the method described above when executed by a processor.

[0023] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0024] The present invention also discloses a 3D multi-target tracking system, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0025] Compared with the prior art, the advantages of the present invention are: The present invention starts with the preprocessing of point cloud data, extracts features and generates a heatmap through a deep learning model, and then ensures the stability of the system in a dynamic environment through efficient multi-target tracking and innovative position and orientation compensation mechanisms; finally, through confidence hierarchical association and trajectory management, the accuracy and robustness of multi-target tracking are further improved.

[0026] The present invention deeply integrates the principle of spatiotemporal interactive joint detection, and realizes the refined processing and feature extraction of the original point cloud data through a carefully designed data processing flow; it not only includes an efficient feature extraction mechanism that can accurately capture the unique identification information of the target, but also incorporates motion compensation technology to cope with the challenges brought by the movement of the self-vehicle and the target vehicle; in addition, it also adopts a confidence hierarchical association strategy, which significantly improves the tracking accuracy and stability by performing multi-level evaluation and association of the confidence of the target tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of a 3D multi-target tracking method according to an embodiment of the present invention.

[0028] Figure 2 The figure is a flow chart of the 3D multi-target tracking method of the present invention in a specific application.

[0029] Figure 3 Schematic diagram of the misalignment of target information of two frames of point clouds at different times in the Lidar coordinate system in the present invention.

[0030] Figure 4 Schematic diagram of target speed in the Lidar coordinate system of the present invention.

[0031] Figure 5 Schematic diagram of target position deviation in the World coordinate system of the present invention.

[0032] Figure 6 Schematic diagram of Lidar coordinate position compensation for detection head velocity in the present invention.

[0033] Figure 7 Schematic diagram of the Lidar coordinate target orientation deviation in the present invention.

[0034] Figure 8 This is an example diagram of Heatmap confidence mean calculation in the present invention.

[0035] Fig. 9 Schematic diagram of the hierarchical multi-level processing of heat maps in the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, the 3D multi-target tracking method of an embodiment of the present invention includes the following steps: S1. Data preprocessing: First, the input vehicle raw point cloud data is preprocessed, including denoising, filtering and other operations to eliminate interference factors, improve data quality, and ensure that the point cloud data used in subsequent processing is cleaner and more accurate; in addition, data enhancement techniques are also used, such as random rotation, translation, scale transformation, random point discarding and other operations, to further enhance the expression ability of the point cloud.

[0038] S2. Deep learning feature extraction and heatmap generation: First, the pre-processed point cloud data is deeply analyzed and features are extracted through a deep learning model. The model uses a multi-layer neural network architecture to gradually extract key features from each frame of point cloud data. These key features can reflect the geometric shape, motion state, and spatial distribution information of the target. After feature extraction, the system generates a corresponding Heatmap to intuitively display the position distribution and feature density of the target in 3D space. Heatmap not only captures the distribution pattern of different targets in the point cloud data, but also contains rich target-related information, such as the size, shape, and dynamic changes of the target in space.

[0039] The key features refer to the multi-dimensional information extracted from each frame of point cloud data through the deep learning model, including the point cloud distribution pattern in space and key data related to the target. The heatmap generated by processing these features not only shows the spatial arrangement of the point cloud, but also captures important information such as the position and shape of the target, becoming an indispensable core foundation in the multi-target tracking process. Specifically, these key features cover the following aspects: first, geometric features. The geometric shape and distribution pattern in the point cloud can help the system identify the outline and relative position of the object; second, motion features. By analyzing multi-frame point cloud data, the system can capture the motion trajectory and dynamic changes of the object; finally, depth and spatial features. The depth information and relative position of the point cloud are the key to forming a 3D structure, which can help distinguish the front and back relationship of the object and its precise positioning in space. Through deep learning mining, these features are converted into important basic data for the target tracking system, supporting the system to achieve multi-target tracking efficiently and accurately.

[0040] This process lays a solid foundation for subsequent multi-target tracking, ensuring that the system can accurately perceive and associate multiple targets, improving the accuracy and stability of target detection.

[0041] S3. Construction of multi-target tracking module: The system builds an efficient multi-target tracking module by combining the Heatmap extracted from deep learning features. This module can seamlessly associate the Heatmap of multiple point clouds in the Lidar coordinate system to achieve accurate joint detection and tracking of multiple targets.

[0042] In order to improve the processing ability of time series, the module introduces an iterative mechanism to combine the real-time information of the current frame with the accumulated data of the historical frame to simulate a memory effect similar to LSTM (Long Short Term Memory). This mechanism ensures that the system can effectively capture the dynamic changes of the target at different times, especially when dealing with complex dynamic environments, and can better adapt to the changes in the target's motion trajectory and the real-time requirements of the scene. This mechanism aims to achieve dynamic tracking of the target by combining the data of the past frame with the information of the current frame. The LSTM model is characterized by its ability to handle long-term dependencies in time series tasks, retain historical information through memory units, and dynamically adjust the state. Although this method does not directly use the LSTM model, it simulates a similar memory effect by introducing an iterative mechanism to combine historical data with current data. This mechanism not only retains the historical trajectory of the target, but also uses the characteristics of the current frame for adjustment, thereby improving the system's adaptability to dynamic changes. During the tracking process, this method continuously fuses multi-frame information, allowing the system to effectively deal with complex situations such as target occlusion and movement, and maintain the continuity and accuracy of tracking. This interaction and accumulation of spatiotemporal information is similar to the memory and update process in LSTM, which makes the tracking system more stable and robust in dynamic environments. In this way, the system can accurately track multiple targets, maintain the continuity and robustness of tracking, and reduce tracking interruptions or deviations caused by target movement or occlusion.

[0043] Specifically, existing multi-target tracking is mostly performed in the TBD (Tracking By Detection) paradigm, where the detection module outputs target information as an upstream task of the tracking module, and the multi-target tracking module assigns track attributes to the detected target information through matching association and other methods. This method results in a one-way effective state between detection and multi-target tracking, and the detection module cannot make good use of the information obtained from multi-target tracking to improve the accuracy of target detection.

[0044] In this regard, the present invention constructs a mathematical model for joint detection of multi-target tracking, analyzes the input and output information of joint detection and tracking; and explains the 3D multi-target tracking network model of joint detection based on deep learning, and analyzes the network model structure and key points, specifically using laser radar as a carrier, and selecting joint detection of 3D multi-target tracking as the research object. In order to clarify the objectives of this task and the content of the original data and final results, a mathematical model for joint detection of multi-target tracking is constructed, and the main data flow of the present invention is described through the mathematical model.

[0045] The laser radar continuous point cloud data is used as input information, and the target recognition and target tracking of the laser radar point cloud data are finally output. The present invention realizes the detection and tracking tasks simultaneously through one model.

[0046] Record the input information of continuous laser point cloud , where P represents the set of laser point clouds in the current scene in the time domain; Represents at the moment Corresponding laser point cloud data, each laser point cloud data consists of composition, represents the position of the laser point cloud, Represents intensity information.

[0047] The output information of joint detection and tracking is recorded as , where O represents the detection and tracking results of continuous input point cloud data; Represents at the moment Corresponding The detection and tracking information of each target is mainly composed of etc., among which Represents the spatial position and size information of the target, Represents the target's orientation angle and velocity information in the corresponding coordinate system. Represent the target category information, tracking ID and confidence information respectively. Goal Represents at different times The following are all real external obstacles.

[0048] Compared with the joint detection multi-target tracking method that uses pixel information as input, the point cloud data obtained by laser radar scanning is used as input. It is necessary to think about how to improve the feature extraction ability of the model and how to better use the extracted information as the basis for joint target detection and multi-target tracking. The model architecture uses continuous multi-frame point clouds as input information, and then extracts features from the original point cloud, generates heat map information at the corresponding moment, compensates for heat map motion in the time domain, and finally outputs the detection and tracking results of the target.

[0049] For example, Figure 2As shown, the first frame of point cloud data is input at time T1, and the point cloud data first obtains higher-dimensional point cloud features through feature extraction. The higher-dimensional point cloud features obtain information such as the target center point according to the decoder, and the target detection result of the first frame of point cloud is obtained according to the center point and other information, and the track information in the current scene is initialized according to the target information of the first frame; when the point cloud at time T1 completes target detection and multi-target tracking, the corresponding tracking results are stored, and the tracking results are motion compensated in the form of a heat map, which is matched with the heat map generated by the features obtained from the input point cloud at frame T2, and the detection and multi-target tracking results are obtained by matching; the above operations on the point clouds of frames T1 and T2 are repeated to obtain the target detection and multi-target tracking results of the point clouds at other subsequent times.

[0050] Motion compensation mechanism: The motion compensation mechanism is designed to solve the problem of Heatmap misalignment caused by the movement of the ego vehicle or target vehicle, and ensure the continuity and accuracy of multi-target tracking. This mechanism accurately calculates the speed information of the target in the Lidar coordinate system, combined with the uniform motion model, to compensate the position of the center point of the Heatmap in real time. When the vehicle or target is moving, the system predicts the next moment position of the target based on the current speed and direction of movement of the target, and corrects the displacement deviation caused by the movement. In this way, the misalignment of the target between different frames is effectively corrected, the tracking trajectory is kept continuous, and the accumulation of errors caused by movement is avoided. Through this motion compensation mechanism, the system can accurately track targets in high-speed or complex dynamic environments, significantly improving the accuracy and stability of multi-target tracking.

[0051] Specifically, when the ego vehicle and the target vehicle are in motion, the target position in the Lidar coordinate system may be offset, resulting in the misalignment of the Heatmap and affecting the tracking accuracy. In order to solve this problem, the present invention accurately calculates the speed information of the target in the Lidar coordinate system, and combines it with the uniform motion model to compensate the position of the center point of the Heatmap. This means that in each frame, the system predicts and corrects the spatial position of the target according to the speed and direction of movement of the target, eliminating the displacement deviation caused by movement. Through this compensation mechanism, the tracking system can not only maintain the continuity of the target, but also improve the accuracy of tracking in a dynamic environment, ensuring accurate multi-target tracking in high-speed or complex motion scenes), effectively eliminating the displacement deviation caused by movement, and ensuring the continuity and accuracy of tracking.

[0052] This module takes the heat map corresponding to the target tracking in the previous frame as input information. Due to the movement of the vehicle and other vehicles, motion compensation is performed before the tracking detection heat map matching process. The target's heat map center point and direction are corrected through the CV (Computer Vision) model through the speed information obtained to prepare for subsequent matching.

[0053] Based on the motion compensation of the target in the current Lidar coordinate system, during the motion process, due to the motion of the ego vehicle and the target vehicle, the target vehicle is often in different positions in the ego vehicle Lidar coordinate system at different times. However, this paper associates the historical point cloud heat map of the previous and next frames based on the current Lidar coordinate system, which strongly depends on the position information of the target in the Lidar coordinate system. Therefore, it is very important to correct and compensate the position of the target in the Lidar. Figure 3 As shown in the figure, the target information of the point clouds in two frames at different times has been greatly misaligned in the Lidar coordinate system.

[0054] In the target Lidar misalignment diagram, the left and right frames of point clouds represent the continuous point clouds of the Tn-1 and Tn frames respectively. The circles of the same color in the left and right figures represent the same object. For example, the green special vehicle and the car in the red circle in the right figure have both moved down significantly relative to the position of the previous moment Tn-1 in the Lidar coordinate system; the vehicle represented by the blue circle in the right figure has not changed much from the position of the previous moment Tn-1 in the Lidar coordinate system. Analysis shows that due to the movement of the self-vehicle LiDAR and the movement of the external target vehicle, there will be an unfixed offset in the coordinates of the point clouds of two consecutive frames.

[0055] In the present invention, the displacement deviation between the previous and next frames is eliminated mainly through a uniform motion model.

[0056] Spatial position and orientation compensation: The spatial position and orientation compensation mechanism aims to solve the problem of the position difference between the target in the Lidar coordinate system and the World coordinate system and the change of the target orientation angle with movement.

[0057] First, spatial position compensation uses the velocity information of the detection head and the velocity data calculated from the tracking results to accurately adjust the position of the target in the two coordinate systems to ensure that the target position in different coordinate systems remains consistent.

[0058] Secondly, orientation compensation dynamically corrects the orientation information in the Heatmap based on possible orientation changes of the target during movement.

[0059] Through this compensation mechanism, the system can accurately track targets in complex dynamic environments, not only ensuring the precise positioning of targets in space, but also ensuring that the targets are always pointed in the correct direction, thereby further improving the accuracy and stability of multi-target tracking. This dual compensation mechanism effectively solves the problems of coordinate misalignment and orientation deviation caused by vehicle movement, making the tracking results more reliable.

[0060] Specifically, speed information is usually a perception result that is difficult for the detection module to directly complete, but in the field of multi-target tracking, it can easily calculate its speed information by analyzing the position information of the same target at different time points. This calculated speed information can better explain the movement pattern of the current target within the previous moment and the movement trend within a certain period of time in the future, thereby achieving more accurate target prediction information and improving the accuracy and robustness of target perception.

[0061] like Figure 4 As shown in the figure, the red arrow represents the X-axis of the Lidar, the green arrow represents the Y-axis of the Lidar, and O represents the origin of the Lidar coordinate system. The blue arrow and the purple arrow represent the speed of the target in the world coordinate system and the speed information of the Lidar in the world coordinate system respectively. The speed in the world coordinate system only represents the movement speed information of the target itself. This speed is calculated by projecting the target in the reference world coordinate system and calculating the position deviation.

[0062] like Figure 5 The figure shows the target position deviation in the World coordinate system.

[0063] The position deviation and speed calculation formula of the tracking result target are:

[0064]

[0065] in, Indicates the target The offset between two frames on the axis, Indicates the target The offset between two frames on an axis; Indicates the target The offset between two frames on the axis; V represents the actual speed information of the final target in the reference world coordinate system, Indicates the time difference between two frames of point cloud that produces the position offset.

[0066] In step S3, target position compensation is performed using the detection head speed and tracking speed in the Lidar coordinate system, and target position compensation is performed in the World coordinate system. , Tracking speed in Lidar coordinate system And the tracking speed in the World coordinate system Through these three types of speed information, the target spatial position compensation of the point cloud thermal map is achieved through different speeds and their combinations in the time domain. Specifically: 1) Detection head speed Compensate the target position in the Lidar coordinate system: In the subtask of the detection head, output the speed information in two directions. These two speeds are respectively , In the tracking result of the previous frame, the target space position is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame;

[0067] in, Represents the updated position information of the previous frame after motion compensation in the Lidar coordinate system; Indicates the location information of the target's detection and tracking results obtained in the previous frame; , They represent the detection head velocity information and the time difference between two frames of point cloud respectively.

[0068] 2) If Figure 6 Shown is the tracking speed in the Lidar coordinate system Compensation for target position: The tracking speed in the Lidar coordinate system is similar to the position compensation of the target in the Lidar coordinate system by the speed output by the detection head. The speed information obtained by the tracking result in the Lidar coordinate system is used to compensate the detection and tracking result heat map of the previous frame:

[0069] in, Represents the detection and tracking result of the previous frame, which is calculated to obtain the target's moving speed in the Lidar coordinate system.

[0070] 3) Tracking speed in the World coordinate system Compensate the target position: To compensate the target position in the World coordinate system, you first need to transfer the target in the Lidar coordinate system to the World coordinate system. The transfer method is shown in the following formula:

[0071] in Represents the coordinate value of the previous frame under World, where the coordinate value is as follows:

[0072] Among them, x, y, z represent the position of the laser point cloud.

[0073] As the The rotation and translation matrix of the frame represents the rotation and translation matrix of the current frame point cloud from the Lidar coordinate system to the World coordinate system, and its size is The specific expression is:

[0074] in, and Represent the corresponding rotation matrix and translation matrix respectively, where is a size of The matrix of is a size of ; 0 in the above formula represents a 1 The zero vector of , 1 represents 1 The unit array.

[0075] The target position and other information in the Lidar coordinate system obtained by feature extraction are calculated by the above formula to obtain the spatial position expressed in the World coordinate system. The speed information position compensation of the target obtained by the tracking result in the World coordinate system is calculated by the above formula:

[0076] In the formula, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system.

[0077] After completing the position compensation, the target position information in the updated World coordinate system needs to be returned to the coordinate system of the current frame, as shown in the following formula:

[0078] In the formula, Represents the inverse of the rotation and translation matrix of the current frame point cloud from the Lidar coordinate system to the World coordinate system. Through the above content, the new position information of the Lidar target in the Lidar coordinate system after the speed information position compensation in the World coordinate system is obtained as the input for the next matching with the current frame.

[0079] 4) Compensation of target position based on confidence-based detection head speed and tracking speed in the Lidar coordinate system: The first three position compensations are all achieved through a single speed information. This method has high requirements for single speed information and requires it to have extremely high accuracy, but usually because the speed calculation of the current frame is performed through the displacement deviation between the previous frame, the calculated speed has a certain lag. At the same time, since the speed information of the detection head is predicted through deep learning, the obtained speed information is also positively correlated with the confidence of the target. Therefore, the predicted speed of the detection head is combined with the speed obtained by tracking in the Lidar coordinate system through the following formula:

[0080] In the formula, Scores represents the speed predicted by the detection head The confidence value of the detection head is given a higher weight value for the predicted speed by the detection head. The lower confidence value indicates that the predicted speed of the detection head is less reliable to a certain extent, which may be caused by occlusion and other reasons. Therefore, a higher weight value is given to the speed in the Lidar coordinate system obtained by tracking. Finally, the composite speed in the Lidar coordinate system is applied to the compensation of the displacement through the following formula to obtain the updated position coordinates: .

[0081] The above introduces the compensation of the spatial position of the target. The orientation information of the target is also extremely important in the perception result. Between the two frames of point cloud, the orientation of the target in the Lidar coordinates may change due to the movement of the vehicle or other vehicles. Therefore, after compensating the target position, it is also extremely important to correct the orientation of the target.

[0082] like Figure 7 As shown in the figure, in the point cloud coordinate system at two moments, the orientation angle of the target has a certain deviation due to its own turning and the movement of the vehicle.

[0083] Through the above analysis, target orientation compensation is mainly divided into two parts: orientation angle change caused by the movement of the vehicle and angular velocity information of other vehicles. The specific calculation formula is as follows:

[0084] In the formula, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; Represents the angular velocity of the corresponding target, which is calculated as follows:

[0085] In the formula, due to the The angular velocity of the target at time Indicates The orientation angle at the moment; Represents the rotation caused by the vehicle's motion, which is obtained by the rotation matrix of the Lidar coordinate system to the World coordinate system of the previous and next two frames. The specific calculation formula is:

[0086] in, , Respectively represent the rotation matrix from the previous moment Lidar coordinates to the current frame World coordinates The element value of , through these two elements to calculate its arc tangent to get the corresponding angle information, can get the angle deviation value caused by the vehicle. The calculation is obtained by the formula:

[0087] In the formula, and They represent the inverse of the rotation matrix from the current frame Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame Lidar coordinate system to the World coordinate system respectively.

[0088] S4. Confidence hierarchical multi-level association strategy: The confidence hierarchical multi-level association strategy aims to improve the accuracy and robustness of target tracking, especially to effectively reduce the risk of mistracking and missed tracking in complex scenarios.

[0089] like Figure 8 The following is an example of calculating the Heatmap confidence mean. The strategy first evaluates the confidence of the targets in the Heatmap, carefully layers the targets with different confidence levels, and combines the high- and low-level target association strategy of the Bytetrack algorithm. For targets with higher confidence levels, a direct association method is used for tracking; for targets with lower confidence levels, a more sophisticated multi-level association strategy is used for processing, ensuring that even under low confidence levels, the system can accurately match and track targets, effectively reducing the risk of mistracking and missed tracking, and significantly improving the performance of the entire tracking system.

[0090] This strategy combines the high- and low-score target association mechanism of the Bytetrack algorithm, further enhancing the system's tracking capabilities in situations with occlusion, target overlap, or complex environments. Through this layered processing, the system can dynamically adjust the association method, making the entire tracking process more accurate and stable, greatly improving tracking performance, especially in multi-target scenarios.

[0091] Specifically, Bytetrack believes that detection boxes below the set threshold will not be used for ID matching, which results in boxes with low confidence being simply discarded due to occlusion or sparse target point clouds, resulting in large blocks of missed detection and track fragmentation. Therefore, Bytetrack proposes a simple and efficient data association method. In simple terms, the idea is to perform multi-stage association on high- and low-scoring targets, and extract some low-scoring targets to achieve more efficient performance.

[0092] like Fig. 9 As shown, the multi-level association method based on confidence level in the present invention is specifically as follows: First, we extract the features of the current frame to get T The heat map information with confidence higher than M is retained; For the input T The heat map information is obtained by extracting the feature of the frame point cloud, and the obstacles are compensated to the current frame T through motion compensation. The Lidar coordinate system is the same as T The frame with a heat map higher than the threshold M is matched. During the matching process, the corresponding position confidence of the heat maps of the previous and next two frames is processed to obtain weighted heat map information. The updated heat map information is divided into the current frame T and the T The registered heat map information and the unregistered T Heatmap information and unregistered T Frame heatmap information.

[0093] The current frame T T The registered heat map information indicates that the current frame successfully matches the historical trajectory to obtain the ID information; the unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, and the track information is not matched; the unmatched track information will not be deleted actively, but the long-term unmatched track information will be eliminated according to the characteristics of the half-decay of the matching weight. The frame heat map information represents the last frame T There is a lack of track heatmaps that can match the current frame, that is, new target information. A higher confidence level N is set for the establishment of new target tracks. It is believed that the target that can obtain a higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for the current frame T above the threshold N, For the heat map information that is not registered above the threshold N, a new track is created directly. For the heat map information below the threshold N, it is considered that no registration with the history is obtained and the confidence score is low. It may be a distant target with incomplete features or a model misdetection, and is selected as the heat map information that needs to be discarded.

[0094] S5. Result output and tracking management: The data in the tracking process will be continuously updated and adjusted through confidence and multi-target association, and finally the trajectory and status of each target will be output. The system manages the point cloud data and the trajectory of the target at each moment, dynamically adjusts the status of the target, and ensures the continuity and accuracy of tracking.

[0095] The present invention starts with the preprocessing of point cloud data, extracts features and generates a heatmap through a deep learning model, and then ensures the stability of the system in a dynamic environment through efficient multi-target tracking and innovative position and orientation compensation mechanisms; finally, through confidence hierarchical association and trajectory management, the accuracy and robustness of multi-target tracking are further improved.

[0096] The present invention deeply integrates the principle of spatiotemporal interactive joint detection, and realizes the refined processing and feature extraction of the original point cloud data through a carefully designed data processing flow; it not only includes an efficient feature extraction mechanism that can accurately capture the unique identification information of the target, but also incorporates motion compensation technology to cope with the challenges brought by the movement of the self-vehicle and the target vehicle; in addition, it also adopts a confidence hierarchical association strategy, which significantly improves the tracking accuracy and stability by performing multi-level evaluation and association of the confidence of the target tracking results.

[0097] The embodiment of the present invention further discloses a computer program product, including a computer program, which executes the steps of the above method when executed by a processor. The embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored, which executes the steps of the above method when executed by a processor. The embodiment of the present invention further discloses a computer device, including a memory and a processor connected to each other, on which a computer program is stored, which executes the steps of the above method when executed by the processor.

[0098] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0099] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A 3D multi-target tracking method, characterized in that: Includes steps: S1. Acquire original point cloud data of the vehicle and perform preprocessing to obtain preprocessed point cloud data; S2, extracting features from the preprocessed point cloud data, obtaining target detection results and generating a heat map; S3, motion compensation is performed on the spatial position and orientation of the target by associating the heat map of multiple frame point clouds; S4. Perform confidence evaluation on the corrected target in the heat map and adjust the detection and tracking strategy based on the evaluation results.

2. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, according to the detection head speed Motion compensation is performed on the target space position in the Lidar coordinate system. Specifically, in the tracking result of the previous frame, the target space position is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame; in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

3. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, the tracking speed in the Lidar coordinate system is Motion compensation is performed on the spatial position of the target. Specifically, in the tracking result of the previous frame, the spatial position of the target is compensated by the following formula to obtain the compensated position information as the matching object of the point cloud heat map of the current frame; in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

4. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, according to the tracking speed in the World coordinate system Motion compensation is performed on the spatial position of the target, specifically: Transfer the target position in the Lidar coordinate system to the World coordinate system to obtain the speed information position compensation of the target in the World: In the formula, Indicates the target position information of the previous frame after position compensation is completed in the World coordinate system; Represents the target coordinate value in the World coordinate system of the previous frame; Represents the time difference between two frames of point cloud; After completing the position compensation, the target position information in the updated World coordinate system is returned to the Lidar coordinate system of the current frame.

5. The 3D multi-target tracking method according to claim 1, characterized in that: In step S3, the detection head velocity based on the confidence in the Lidar coordinate system is Tracking speed Motion compensation for the target spatial position is performed by detecting the head speed Tracking speed in Lidar coordinate system Combine to get the composite speed : In the formula, Scores represents the detection head speed The confidence value of the Lidar coordinate system is given a higher weight for the detection head speed, and a lower confidence value for the tracking speed in the Lidar coordinate system. Assign a higher weight value; The composite velocity in the Lidar coordinate system is applied to the displacement compensation through the following formula to obtain the updated position coordinates: in, Represents the updated position information of the previous frame after motion compensation; Indicates the location information of the target's detection and tracking results obtained in the previous frame; Represents the time difference between two frames of point cloud.

6. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S3, the specific formula for motion compensation of the target orientation is: In the formula, Indicates the result of the target heading angle of the previous frame after heading angle offset compensation; Indicates the orientation angle information of the target detection and tracking result of the previous frame; Represents the time difference between two frames of point cloud; Indicates that the corresponding target is The angular velocity at the moment is calculated as: In the formula, Indicates The orientation angle at the moment; Represents the rotation caused by the vehicle's motion, which is obtained by the rotation matrix of the Lidar coordinate system to the World coordinate system of the previous and next two frames. The specific calculation formula is: in, , Represents the rotation matrix from the previous moment's Lidar coordinate system to the current frame's Lidar coordinate system The element value of , the arc tangent is calculated through these two elements to obtain the corresponding angle information, that is, the angle deviation value caused by the vehicle; in The calculation is obtained by the formula: In the formula, and They represent the inverse of the rotation matrix from the current frame Lidar coordinate system to the World coordinate system and the rotation matrix from the previous frame Lidar coordinate system to the World coordinate system respectively.

7. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S4, the specific process of adjusting the detection and tracking strategy according to the evaluation results is as follows: First, we extract the features of the current frame to get T The heat map information with confidence higher than the threshold M is retained; For the input T Frame point cloud feature extraction to obtain T The heat map information is used to compensate the obstacles to the current frame T through motion compensation. The Lidar coordinate system is the same as T The heat map of the frame above the threshold M is matched; in the matching process, the corresponding position confidence of the heat map of the previous and next two frames is processed to obtain the weighted heat map information; the updated heat map information is divided into the current frame T according to the confidence. T The registered heat map information and the unregistered T Heatmap information and unregistered T Frame heat map information; The current frame T T The registered heat map information indicates that the current frame successfully matches the historical trajectory to obtain ID information; Unregistered T The heat map information represents the current frame T There is no target that can match the historical trajectory, that is, unmatched track information; the unmatched track information will not be deleted actively, but the long-term unmatched track information will be eliminated according to the characteristics of the half-decay of the matching weight; Unregistered T The frame heat map information represents the last frame T There is a lack of track heatmaps that can match the current frame, that is, new target information; a confidence level N is set for the establishment of new target tracks, and it is believed that the target that can obtain a higher confidence level in the current frame has sufficient ability to be a new track. Therefore, for unregistered T above the threshold N, The frame heat map directly creates a new track. For heat map information below the threshold N, it is considered that the registration with the history is not obtained and the confidence score is low. It is a distant target with incomplete features or a model misdetection, and is selected as the heat map information that needs to be discarded.

8. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S2, the features in the point cloud data include geometric features, motion features, depth features and spatial features.

9. The 3D multi-target tracking method according to any one of claims 1 to 5, characterized in that: In step S1, the preprocessing includes denoising, filtering and data enhancement.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are performed.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 9.

12. A 3D multi-target tracking system, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 9.

Images