3D multi-target tracking method based on probability distribution and state regression

Abstract

The present invention relates to a 3D multi-target tracking method based on probability distribution and state regression, and belongs to the field of environmental perception. It includes: taking the point cloud data and historical trajectory at the current moment as input, generating the 3D detection at the current moment and the state of the historical trajectory at the current moment through the 3D detection and trajectory regression module, wherein, in the regression processing process, first extracting the features of the point cloud at the current moment and then inputting it into the RPN module to generate a region candidate frame; at the same time, using the Kalman filter in combination with the vehicle posture to predict and compensate the historical trajectory state; then obtaining the corresponding prior candidate frame set through probability grid sampling according to the uncertainty of each trajectory state; then merging the region candidate frame with the prior candidate frame set; then sending the merged data to the voxel RoI Pooling module for feature alignment; sending the aligned features to the detection head, and outputting the final trajectory after screening and trajectory management.

Description

Technical Field

[0001] The invention belongs to the field of environmental perception and relates to a 3D multi-target tracking method based on probability distribution and state regression. Background Art

[0002] 3D multi-target tracking is widely used in autonomous driving, robot navigation, intelligent monitoring and other fields by determining the positions of different objects in consecutive frames and obtaining dynamic information of the surrounding environment. In the past few years, MOT can be divided into different paradigms according to the tracking method: tracking by detection (TBD), joint detection and embedding (JDE), tracking by attent-ion (TBA) and joint detection and tracking (JDT), which greatly promoted the development of MOT technology.

[0003] At present, MOT based on the TBD paradigm has become the preferred paradigm for solving tracking problems. It decomposes the task into two steps: 1) target detection; 2) data association. First, the 3D detector is used to detect the target in the current frame image; second, the state of the trajectory of the previous frame is predicted; finally, the predicted trajectory is used to perform complex data association with the current detection. In addition, the MOT method of JDE adds a feature extraction branch to the detector network to implement target detection and feature extraction in parallel. Compared with the MOT of TBD, the MOT of JDE reduces the inference time, but the complex data association strategy is still unavoidable. However, due to missed detection, occlusion and incorrect association competition between targets, it is easy to cause mismatching between detection and trajectory. In order to solve this problem, some MOT methods design new Kalman filters to improve the accuracy of trajectory prediction, and other MOT methods achieve more robust matching by designing complex cost functions. However, no matter how to improve the prediction accuracy and data association accuracy, since detection and tracking are performed independently in the TBD paradigm, the prior information of the trajectory is not used to guide tracking, and end-to-end multi-target tracking cannot be achieved.

[0004] When the trajectory is predicted continuously, the accumulation of accumulated errors will cause the trajectory state to be unstable. However, state regression inputs the historical trajectory frame into the two-stage target detector, which can make full use of the detection features of the point cloud and solve the problem of the inability to update the state caused by continuous state prediction to a certain extent. In the previous regression framework, the trajectory frame of the historical frame was input into the two-stage detector as a candidate frame for regression, and NMS was used to achieve trajectory update and trajectory regeneration, further improving the position accuracy of the trajectory. However, the previous regression only performed state regression at a fixed position and did not perform a range search in the potential detection space. When the trajectory state is unstable, this regression method also has the limitation of inaccurate position update. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a 3D multi-target tracking method based on probability distribution and state regression. The present invention introduces an end-to-end multi-target tracking method based on sampling regression and trajectory interaction. The method performs sampling regression based on probability distribution and accurately regresses the trajectory of the current frame; based on trajectory interaction, it suppresses the trajectory of erroneous regression and reduces the matching competition of regression update. The method can achieve robust 3D multi-target tracking with only one 3D detector.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A 3D multi-target tracking method based on probability distribution and state regression includes: taking the point cloud data at the current time t and the historical trajectory at the time t-1 as input, generating the 3D detection at the current time t and predicting and regressing the state of the historical trajectory at the time t-1 at the current time t through the 3D detection and trajectory regression module, wherein in the regression process, the point cloud at the current time t is feature extracted, and these features are input into the RPN module to generate a region candidate frame. At the same time, the Kalman filter is used to combine the posture information of the vehicle to predict and compensate the state of the historical trajectory at the time t-1. Next, according to the uncertainty of each trajectory state, the corresponding prior candidate frame set is obtained through the probability grid sampling technology. Subsequently, the region candidate frame generated by the RPN module is merged with the prior candidate frame set obtained by the probability grid sampling. The merged data is sent to the voxel RoIPooling module for feature alignment. The aligned features are further sent to the detection head to perform state regression, thereby completing the 3D detection and trajectory state update. After all the regressed trajectories are obtained, they are screened to identify stable trajectories. At the same time, the 3D detection results are screened to identify and mark the new trajectories. Finally, the trajectory management module is used to uniformly control and manage these trajectories.

[0008] Furthermore, during the probability sampling process, the trajectory state of the probability sampling model is set Tra X=(x,y,z,θ,l,w,h,v x ,v y ,v z ) T is a 10-dimensional trajectory state space, where (x, y, z represent the position of the target on the x-axis, y-axis, and z-axis respectively, θ is the orientation angle of the target, (v x ,v y ,v z ) represent the speed of the target on the x-axis, y-axis and z-axis respectively; the state (x, y, z, θ, l, w, h) needs to be updated by regression, and the number of sample points can be reduced by simplifying assumptions, including the scale invariance assumption and the height and position invariance assumption, so only (x, y, θ) needs to be sampled.

[0009] Furthermore, according to the probability sampling model, a probability grid sampling method is adopted for sampling, and the sampling range is adjusted accordingly for different target vehicle postures based on the tracking target's own coordinate system; combined with the statistical law of Gaussian distribution, for different trajectories, the sampling ranges |Δx| and |Δy| are determined using the following formula:

[0010] |Δx|=min(2σ x ,r x )

[0011] |Δy|=min(2σ y ,r y )

[0012] In the formula, σ x and σ y are the standard deviations in the x and y directions of the trajectory state; r x and r y They are the sampling thresholds in the x and y directions respectively.

[0013] When sampling the target orientation angle θ, after obtaining the trajectory of the target after sampling in the x and y directions, the sampling set of each trajectory is obtained through the bird's-eye view BEV; the regional candidate network RPN is used to adjust the orientation angle of the sampling set, and then the adjusted orientation angle set is mapped to the point cloud grid to correct the orientation angle of the initial sampling set.

[0014] Furthermore, after feature extraction of the point cloud at the current time t is input into the RPN module to generate a region candidate frame, the RPN module obtains the region candidate frame by regressing and classifying the preset anchor frame.

[0015] Furthermore, the process of predicting and compensating the historical trajectory state at time t-1 by combining the ego vehicle posture with the Kalman filter includes using the ego vehicle posture to perform motion compensation on the target state and using the Kalman filter to predict the target motion.

[0016] Furthermore, the voxel RoIPooling module for feature alignment includes scale transformation and feature mapping.

[0017] Furthermore, after the 3D detection and trajectory state regression are completed, the trajectory regression state needs to be updated. The historical trajectory obtains the corresponding 3D confidence and 3D state through the 3D regression module, and different adjustment schemes are adopted for the historical trajectory according to the 3D confidence threshold; when the 3D confidence is lower than the threshold, the original predicted 3D state is replaced by the 3D state obtained by regression; when the 3D confidence is greater than the threshold, the Kalman update step is used to adjust its state.

[0018] Furthermore, the propagation of trajectory states is completed through state regression. All trajectories are scanned at each moment through the trajectory module. For mutually occupied trajectories, the confidence of each other's trajectories is compared, and the trajectories with low confidence are removed.

[0019] Furthermore, greedy matching based on trajectory confidence is finally performed to screen out new trajectories, which specifically includes: sorting the trajectories from high to low according to the confidence of the trajectory state at the current moment, and then calculating the associated costs of the trajectory target and all detected targets one by one to determine whether the detected target is a new trajectory;

[0020] Furthermore, the trajectory status is maintained and output control based on trajectory confidence is performed through the post-processing module; wherein the trajectory management strategy is specifically as follows:

[0021] If the trajectory state regression confidence is greater than the update threshold σ1, the trajectory is output to the downstream task. When 12 consecutive frames are less than the update threshold σ1, the trajectory state is converted to a lost state and no longer maintained.

[0022] If the association cost between the detected target and all trajectory targets is less than the new threshold σ2, the detection result will be new at the next moment. If the new trajectory is successfully updated in three consecutive frames, it will be transferred to the determined state.

[0023] The beneficial effects of the present invention are:

[0024] In order to expand the search space of trajectory regression and suppress the overlap between regression trajectories, the present invention innovatively proposes a 3D multi-target tracking method based on probability distribution and state regression. The tracking method proposed in the present invention mainly includes the following parts: First, in order to solve the problem of unstable trajectory state that may be caused by Kalman filtering in continuous prediction, a probability sampling method is designed, which can adaptively expand the regression range. Secondly, the state regression model is used to predict the position of historical targets at the current moment to make full use of the prior information of the trajectory state. Finally, the trajectory interaction module is used to suppress overlapping erroneous trajectories, thereby avoiding erroneous regression of trajectories during continuous prediction.

[0025] The present invention proposes a novel end-to-end multi-target tracking method, which eliminates complex data association, and based on the prior information of the trajectory, accurately regresses the trajectory of the current frame through trajectory sampling regression, thus achieving a more robust trajectory update.

[0026] Compared with the current SOTA MOT method, the MOT method proposed in this invention achieves excellent tracking performance on the KITTI and Waymo datasets.

[0027] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 It is the framework structure of 3D multi-target tracking method based on probability distribution and state regression;

[0030] Figure 2 This is a schematic diagram of the state regression failure scenario;

[0031] Figure 3 This is a schematic diagram of a successful sampling regression scenario;

[0032] Figure 4 This is the improved two-stage detector framework diagram;

[0033] Figure 5 is an example diagram of trajectory interaction, where Figure 5 (a) An example of a deviation from the detection result for correct trajectory regression; Figure 5 (b) An example of a correct detection but incorrect trajectory regression.

[0034] Figure 6 This is a visualization example of the present invention in the KITTI dataset, where Figure 6 (a) is the occlusion scene, Figure 6 (b) is a complex intersection scene. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0037] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0038] See also Figure 1 to Figure 6 , which is a 3D multi-target tracking method based on probability distribution and state regression.

[0039] In the previous multi-target tracking paradigm, in order to match detection and trajectory, it is often necessary to design complex data association strategies. This paper proposes a novel end-to-end 3D multi-target tracking framework - joint detection and tracking based on probability distribution and state regression. First, the framework does not need to rely on complex data association strategies, but uses the prior information of the trajectory to regress the precise position of the trajectory at the current moment. Secondly, a probabilistic grid sampling strategy is adopted to adaptively expand the regression search range of the trajectory, thereby reducing the uncertainty of the trajectory state caused by continuous prediction. Thirdly, in order to solve the overlap between trajectories, a trajectory interaction module is introduced to retain trajectories with higher confidence. Finally, a large number of experiments are carried out on the KITTI and Waymo datasets. The results show that the framework can achieve robust tracking using only one 3D detector and outperforms many SOTA tracking methods.

[0040] Example 1

[0041] This embodiment provides Figure 1 The framework structure of the 3D multi-target tracking method based on probability distribution and state regression shown in the figure includes three parts: data input module, 3D detection and trajectory regression module and data post-processing module.

[0042] The data input part includes the point cloud data at the current time t and the historical trajectory at the time t-1 (the laser radar coordinate system at the time t-1).

[0043] The 3D detection and trajectory regression module contains two main branches. The first branch is responsible for generating 3D detection at the current time t, following the typical two-stage detection process: first, feature extraction is performed, and the point cloud information at time t is input into the 3D network backbone to obtain feature information; then, region candidate boxes are generated, and the extracted feature information is input into the 3D Region Proposal Network (3D RPN) to obtain region candidate boxes, followed by feature alignment through the RoIPooling network, and finally classification and regression operations. The second branch focuses on predicting and regressing the state of the historical trajectory at the current moment. This process first uses the Kalman filter to predict and compensate the trajectory state in combination with the current vehicle pose. Subsequently, the probability grid sampling module samples the corresponding prior candidate box set according to the uncertainty of each trajectory state. At the same time, the RoIPooling module aligns the features and regresses their precise states. In order to avoid mutual interference between objects, the trajectory interaction module is used to filter out some objects with low confidence and overlapping with each other. Finally, greedy matching based on trajectory confidence is performed to screen out new trajectories.

[0044] The post-processing module mainly includes the maintenance of trajectory status (such as dead, pending and confirmed), output control based on trajectory confidence and other post-processing operations. This embodiment adopts the same trajectory management strategy as that in UG3DMOT.

[0045] In this embodiment, the regression update of the state can effectively alleviate the association difficulty problem caused by inaccurate prediction. However, the accuracy of the state regression is highly dependent on the position information of the trajectory. Figure 2 As shown in , when the position prior information is inaccurate, although the trajectory actually exists at the current moment, the state obtained by regression is still unstable, which is manifested in that the trajectory not only cannot obtain a high confidence through regression, but also has obvious deviations in position and size. In response to the above problems and the phenomenon that 3D scene objects cannot overlap with each other, as shown in Figure 3 As shown in the figure, a probability sampling method is used to sample near the mean of the original prediction state, adaptively expand the search space of the trajectory, and obtain more accurate regression results.

[0046] The trajectory state of the probability sampling model in this embodiment Tra X=(x,y,z,θ,l,w,h,v x ,v y ,v z ) T is a 10-dimensional trajectory state space, where (x, y, z) represents the position of the target on the x-axis, y-axis, and z-axis, θ is the orientation angle of the target, and l, w, and h represent the size of the target, i.e., length, width, and height, respectively. x ,v y ,v z ) represent the speed of the target on the x-axis, y-axis and z-axis respectively. Among them, the state of (x, y, z, θ, l, w, h) 7 dimensions needs to be regressed and updated. Due to the large number of dimensions of the multidimensional Gaussian distribution, directly sampling all dimensions will result in too many samples and cannot meet the real-time requirements. Therefore, some assumptions need to be made in the sampling process to reduce the number of sample points. Scale invariance assumption: The length, width and height dimensions are one of the inherent characteristics of the object, so the three dimensions of length, width and height can be regarded as fixed components and do not need to be sampled. Height position invariance assumption: For object detection and tracking, the height position generally does not change significantly within the time interval of ΔT, and the height position at the previous moment can be used directly. Therefore, for the 7-dimensional Gaussian distribution of (x, y, z, θ, l, w, h), after applying the simplified assumption, only (x, y, θ) needs to be sampled. For the tracking task, the x-axis position and y-axis position of the tracked target and the orientation angle θ are constantly changing, and the state uncertainty is strong, so sampling is required.

[0047] Preferably, this embodiment adopts probabilistic grid sampling for sampling. To address the problem of inaccurate trajectory prior information, grid sampling based on the coordinate system of the tracked target itself is used, which can be adjusted accordingly for different target vehicle postures. In the statistical characteristics of Gaussian distribution, about 95.45% of the samples are distributed within two standard deviations around the mean. Based on this statistical law, this embodiment uses 2σ as the sampling threshold. For different trajectories, the sampling ranges |Δxv and |Δy| are determined using the following formula:

[0048] |Δx|=min(2σ x ,r x )

[0049] |Δy|=min(2σ y ,r y )

[0050] In the formula, σ x and σ y are the standard deviations of the x and y directions in the trajectory state. In order to prevent the trajectory from regressing to other objects, the sampling thresholds in the x and y directions are set to r x and r y .

[0051] In addition to sampling the x and y position components, the orientation angle θ also needs to be sampled. The two-stage target detector has an RPN structure, which can perform preliminary regression on the preset anchor frame at each grid position, so the present invention adopts an adaptive angle adjustment method. First, using the trajectory that has been sampled in the x and y directions, a sampling set of each trajectory is obtained from a bird's-eye view (BEV). Then, the regional candidate network (RPN) structure is used to adjust the orientation angle of the sampling set to make it closer to the actual target posture. Next, these sampling sets are mapped to the point cloud grid to correct the orientation angle of the initial sampling set. The advantage of this method is that it can effectively utilize the prior knowledge obtained by the anchor frame in the preliminary regression stage, thereby avoiding additional sampling of the orientation angle. The number of samples is reduced, thereby improving the efficiency of the entire trajectory regression process.

[0052] Preferably, the state regression of the trajectory can effectively combine the prior information of the trajectory, so as to more accurately estimate the state of the trajectory at the current moment. The method proposed in the present invention improves the two-stage point cloud detector to meet the specific requirements of the present invention regarding state regression. The state regression of the present invention adopts a two-stage regression method. In the first stage, the regional candidate network is used to preliminarily screen potential target positions. Then, the detection head in the second stage performs more refined regression processing on these candidate positions. The improved two-stage detector of the present invention realizes parallel processing of 3D detection and historical trajectory regression by integrating the prior box of the historical state trajectory. Figure 4 As shown in the figure, the specific process includes the following steps: First, feature extraction is performed on the current point cloud, and the extracted features are sent to the RPN module to generate a region candidate box. At the same time, motion compensation and Kalman filter prediction are performed on the historical trajectory to obtain updated trajectory information. Subsequently, the region candidate box generated by the RPN module is spliced ​​and merged with the prior candidate box based on the historical trajectory, and then the merged data is sent to the voxel RoIPooling module for feature alignment. Finally, the aligned features are sent to the detection head to perform state regression, thereby completing 3D detection and trajectory state regression.

[0053] The process of extracting features from the current point cloud and sending the extracted features to the RPN module to generate a region candidate frame is specifically to obtain the region candidate frame by regressing and classifying the preset anchor frame.

[0054] Implementing motion compensation and Kalman filter prediction on historical trajectories includes the following steps: using the ego-vehicle posture to perform motion compensation on the target state and using Kalman filter to predict the target motion.

[0055] The voxel RoIPooling module specifically includes scale transformation and feature mapping.

[0056] After motion compensation and Kalman prediction, the state of the target object is still uncertain, especially in the scene of object initialization and turning. After initialization, the object usually needs to go through several updates before its state can gradually stabilize. When the object turns, due to the use of a uniform linear motion model, there must be a deviation between the predicted result and the actual position, which is particularly obvious in the case of low frame rates. Therefore, in order to improve the accuracy of tracking, a state update is required after regression to make up for the deficiencies of the model and improve the accuracy of tracking.

[0057] The historical trajectory is passed through the 3D regression module to obtain the corresponding 3D confidence and 3D state, and different adjustment schemes are adopted for the historical trajectory according to the 3D confidence threshold. When the 3D confidence is lower than the threshold, the original predicted 3D state is replaced by the 3D state obtained by regression for the consideration of more accurate 3D regression; when the 3D confidence is greater than the threshold, the Kalman update step is used to adjust its state to improve its stability.

[0058] The detection head performs two steps: classification and regression. For classification, an MLP structure is used to determine the target category. For regression, the corresponding offset is regressed based on the prior information of the object candidate box, and then the two are superimposed to obtain the final target state.

[0059] Data association helps the trajectory complete the state propagation. For the trajectory that has not been successfully associated, continuous prediction will lead to the gradual divergence of the trajectory state, which may cause the historical trajectory and the corresponding detection to be unable to be associated, resulting in two trajectories pointing to the same object at the next moment. At the same time, the detector outputs a detection box with deviations, which may also cause the two trajectories to occupy each other at the next moment. Figure 5 As shown in the figure, the dotted line represents the trajectory box and the solid line represents the detection box. Figure 5 In (a), the trajectory regression is correct but the detection is biased; Figure 5 In (b), the detection is correct but the trajectory regression is wrong. When the data association fails in these two cases, the detection box will become a new trajectory at the next moment, and the two trajectories point to the same object. Based on the non-overlapping phenomenon of objects in 3D space, the trajectory interaction module is introduced. All trajectories are scanned at each moment, and for trajectories that occupy each other, the trajectories with low confidence are removed.

[0060] The algorithm for screening out new trajectories based on greedy matching of trajectory confidence includes the following: according to the confidence of the trajectory state at the current moment, the trajectories are sorted from high to low according to the confidence, and then the association cost between the trajectory target and all detected targets is calculated one by one to determine whether the detected target is a new trajectory.

[0061] This embodiment adopts the same trajectory management strategy as that in UG3DMOT, including: if a new trajectory is continuously regressed and updated for 3 frames, it is converted into a deterministic trajectory; if a deterministic trajectory is not successfully updated for 12 consecutive frames, the trajectory is deleted.

[0062] Example 2

[0063] This embodiment conducts a large number of experiments on the KITTI and Waymo datasets based on the 3D multi-target tracking method based on probability distribution and state regression proposed in Example 1. The results show that the framework can achieve robust tracking using only one 3D detector and is superior to many SOTA tracking methods.

[0064] This example provides the following experimental environment:

[0065] The MOT method proposed in the present invention is implemented on a desktop computer equipped with Python programming language, Intel Core i7-10700 CPU and 32GB RAM.

[0066] The commonly used datasets used for evaluation in this embodiment include KITTI and Waymo. These two datasets have been widely used in the field of computer vision and are generally considered to be reliable data sources for algorithm evaluation.

[0067] To evaluate the effectiveness of the proposed method, state-of-the-art multi-object tracking algorithms on the leaderboard are employed for comparative study.

[0068] This embodiment uses the evaluation indicators used by the KITTI dataset and the Waymo dataset to evaluate the tracking performance of the above-mentioned competitive methods. These commonly used indicators include but are not limited to CLEAR and HOTA. The HOTA indicator is the main evaluation indicator used by the KITTI dataset, which involves multiple indicators such as detection accuracy (DetA) and association accuracy (AssA). The Waymo dataset mainly uses the CLEAR indicator, and divides CLEAR into two indicators, LEVEL_1 (L1) and LEVEL_2 (L2), according to the difficulty of detecting the target. L1 represents easy-to-detect objects, and L2 represents difficult-to-detect objects.

[0069] This example first uses the KITTI dataset to verify the proposed method. The evaluation results are shown in Table 1.

[0070] Table 1

[0071]

[0072] The method proposed in this embodiment has obvious advantages in HOTA, AssA and MOTP indicators. Specifically, when only using 3D point cloud information, the method of this embodiment achieves the highest HOTA of 78.23% among many methods including multimodal fusion; for the MOTP indicator that evaluates trajectory position accuracy, it achieves the best value of 86.91% among all the methods in the table. In terms of the AssA indicator, the method of this embodiment leads all other competing methods with a score of 82.01%. This indicator focuses on evaluating the association ability of the algorithm.

[0073] In addition to the commonly used KITTI dataset, another large dataset, the Waymo dataset, is also used for further evaluation. The evaluation results are shown in Table 2, which also show excellent performance.

[0074] Table 2

[0075]

[0076]

[0077] Compared with Waymo's baseline method PK-baseline, the method proposed in this embodiment has a significant improvement in MOTA (L1 / L2) by 32.35% and 31.93% respectively. Compared with other latest methods, such as PV-RCNN-KF, the method proposed in this embodiment also has significant improvements in MOTA (L1 / L2). Figure 6 As shown, two fragment scenes of the 0010 sequence and the 0014 sequence in the test dataset of the KITTI dataset are used for visualization. Given that images are conducive to showing occlusion and missed detection, the first four rows of the scene are object boxes formed by projecting 3D objects onto the image. The first and fourth rows of each scene show the starting frame and the ending frame of the fragment scene, respectively, while the second and third rows show the missed detection and occlusion of objects, and the dotted boxes indicate missed objects or occluded objects. In addition, the fifth row shows a comprehensive view of projecting the historical information of the trajectory onto a specific frame, and the dotted box indicates a missed object or an occluded object. In addition, the fifth row shows a comprehensive view of projecting the historical information of the trajectory onto a specific frame, and the dotted elliptical box indicates that there are missed historical trajectory points.

[0078] exist Figure 6 In the scene shown in (a), when encountering the occlusion problem, a vehicle (ID 132) that is in the process of turning turns right, while the two vehicles behind it (ID 129 and 126) continue to go straight. During frames 235 to 245, the vehicle with ID 132 cannot be detected due to the occlusion of the vehicle behind it. Even in such a complex occlusion situation, the method introduced in this embodiment can still accurately associate the correct vehicle, demonstrating its powerful performance in handling occlusion scenes. Figure 6 (b) shows that in a complex intersection scene, vehicle ID 10 passes in front of vehicle ID 12 at frame 29, causing a brief occlusion. After several frames of occlusion, vehicle ID 12 reappears. At frame 63, vehicle ID 10 is missed for several frames due to a detector failure. Nevertheless, the comprehensive view of the lidar shows that regardless of occlusion or missed detection, the method of this embodiment can still stably track vehicles ID 12 and ID 10.

[0079] In order to explore the impact of different modules on the overall tracking performance, this example uses the KITTI validation set to conduct an ablation experiment. The evaluation index uses two important indicators: CLEAR and HOTA.

[0080] In order to explore the role of the sampling regression module, this example compares the effects of the three methods on the KITTI dataset verification sequence. As shown in Table 3, the sampling regression method proposed in this example obtains the highest HOTA, DetA, AssA and IDSW indicators.

[0081] Table 3

[0082]

[0083] Specifically, the TBD paradigm has the lowest HOTA index, while the sampling regression method performs the best. The TBD method separates detection and association and cannot utilize the prior information of the trajectory. The ordinary state regression method regresses the object position by utilizing the prior information of the trajectory, while the sampling regression method effectively solves the regression failure problem caused by inaccurate prediction by expanding the regression range, thereby improving the accuracy and stability of tracking.

[0084] In order to explore the role of the trajectory interaction module, this embodiment is tested on the KITTI validation set. As can be seen from Table 4, the use of the trajectory interaction module helps to reduce the false competition of trajectory pair detection and reduce false trajectories (FP).

[0085] Table 4

[0086]

[0087] In order to explore the impact of different detectors on the overall tracking performance, this example uses CasA, Point RCNN, and Voxel R-CNN for ablation experiments. Table 5 shows the comparison of the impact of the three detectors on the trajectory regression module when the CasA detection results are used uniformly.

[0088] Table 5

[0089]

[0090] The voxel grid-based Voxel R-CNN and CasA detectors can achieve the best tracking effect. The point feature-based Point RCNN detector is inferior to the above two detectors, but can also achieve good tracking effect.

[0091] Table 6 shows the experimental results of tracking using the detection results of each detector. Undoubtedly, CasA, as the optimal detector, has the best tracking effect, while Point RCNN has the lowest tracking performance due to its low detection performance.

[0092] Table 6

[0093]

[0094] Comprehensively comparing Table 5 and Table 6, the biggest difference between the CasA detector, Point RCNN detector and Voxel RCNN detector comes from the detection of new objects. Although the three detectors have different processing methods, they can effectively use the prior information of the trajectory to update the trajectory state.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A 3D multi-target tracking method based on probability distribution and state regression, characterized by: It includes: The point cloud data at the current time t and the historical trajectory at the time t-1 are used as input, and the 3D detection at the current time t and the state of the historical trajectory at the current time t are generated through the 3D detection and trajectory regression module; Among them, in the regression processing, the point cloud at the current time t is feature extracted, and these features are input into the RPN module to generate the region candidate frame; the Kalman filter is used to combine the posture information of the vehicle to predict the historical trajectory state at time t-1 and make necessary compensation; next, according to the uncertainty of each trajectory state, the corresponding prior candidate frame set is obtained through the probability grid sampling technology; then, the region candidate frame generated by the RPN module is merged with the prior candidate frame set obtained by probability grid sampling; the merged data is sent to the voxel RoIPooling module for feature alignment; the aligned features are further sent to the detection head to perform state regression, thereby completing 3D detection and updating the trajectory state; after obtaining all the regressed trajectories, they are screened to identify stable trajectories; at the same time, the 3D detection results are screened to identify and mark the new trajectories; finally, the trajectory management module is used to uniformly control and manage the trajectories; During the probability sampling process, set the trajectory state of the probability sampling model Tra X=(x,y,z,θ,l,w,h,v x ,v y ,v z ) T is a 10-dimensional trajectory state space, where (x, y, z) represents the position of the target on the x-axis, y-axis, and z-axis respectively, θ is the orientation angle of the target, (v x ,v y ,v z ) represent the speed of the target on the x-axis, y-axis and z-axis respectively; The state (x, y, z, θ, l, w, h) needs to be updated by regression. The number of sample points can be reduced by simplifying assumptions, including the scale invariance assumption and the height and position invariance assumption. Then, only (x, y, θ) needs to be sampled. According to the probability sampling model, the probability grid sampling method is used for sampling. The sampling range is adjusted accordingly for different target vehicle postures based on the tracking target's own coordinate system. Combined with the statistical law of Gaussian distribution, for different trajectories, the sampling range |Δx| and |Δy| are determined using the following formula: In the formula, σ x and σ y are the standard deviations in the x and y directions of the trajectory state; r x and r y are the sampling thresholds in the x and y directions respectively; When sampling the target orientation angle θ, after obtaining the trajectory of the target after sampling in the x and y directions, the sampling set of each trajectory is obtained through the bird's-eye view BEV; the regional candidate network RPN is used to adjust the orientation angle of the sampling set, and then the adjusted orientation angle set is mapped to the point cloud grid to correct the orientation angle of the initial sampling set.

2. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: After feature extraction of the point cloud at the current time t, the feature extraction is input into the RPN module to generate the region candidate frame. The RPN module obtains the region candidate frame by regressing and classifying the preset anchor frame.

3. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: The process of predicting and compensating the historical trajectory state at time t-1 by combining the ego vehicle posture with the Kalman filter includes using the ego vehicle posture to perform motion compensation on the target state and using the Kalman filter to predict the target motion.

4. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: The voxel RoIPooling module for feature alignment includes scale transformation and feature mapping.

5. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: After 3D detection and trajectory state regression are completed, the trajectory regression state needs to be updated. The historical trajectory obtains the corresponding 3D confidence and 3D state through the 3D regression module, and different adjustment schemes are adopted for the historical trajectory according to the 3D confidence threshold; when the 3D confidence is lower than the threshold, the original predicted 3D state is replaced by the 3D state obtained by regression; when the 3D confidence is greater than the threshold, the Kalman update step is used to adjust its state.

6. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: The propagation of trajectory state is completed through state regression, and then all trajectories are scanned at each moment through the trajectory interaction module. For mutually occupied trajectories, the trajectory confidences are compared with each other, and the trajectories with low confidence are removed.

7. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: Finally, greedy matching based on trajectory confidence is performed to filter out new trajectories from 3D detection. Specifically, the trajectories are sorted from high to low according to the confidence of the trajectory state at the current moment, and then the association cost between the trajectory target and all detected targets is calculated one by one to determine whether the detected target is a new trajectory.

8. The 3D multi-target tracking method based on probability distribution and state regression according to claim 1, characterized in that: The trajectory status is maintained through the post-processing module and the output control based on the trajectory confidence is performed; the trajectory management strategy is specifically as follows: If the trajectory state regression confidence is greater than the update threshold σ1, the trajectory is output to the downstream task; when 12 consecutive frames are less than the update threshold σ1, the trajectory state is converted to a lost state and no longer maintained; If the association cost between the detected target and all trajectory targets is less than the new threshold σ2, the detection result will be generated at the next moment; if the new trajectory is successfully updated in three consecutive frames, it will be transferred to the determined state.

Images