Tracking information processing method, program product, electronic device and chip system

Abstract

This application provides a tracking information processing method, program product, electronic device, and chip system based on laser point clouds, relating to the field of autonomous driving technology. This method can improve the reliability of observations or correct problems of disconnected and misconnected trajectories. The method may include the following steps: determining the confidence level corresponding to at least one first bounding box to be processed; at least based on the confidence level, filtering reliable inputs from at least one first bounding box to obtain a reliable input sequence. Alternatively, the method may include the following steps: determining N trajectories to be processed; sequentially inputting the m-th trajectory into a prediction model to obtain a first trajectory; inputting the first trajectory into the prediction model in reverse order to obtain a second trajectory; the second trajectory is the trajectory obtained by reverse prediction of the first trajectory; and filling in the missing bounding boxes in the m-th trajectory and / or the second trajectory based on the first trajectory and / or the second trajectory.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a tracking information processing method, program product, electronic device and chip system. Background Technology

[0002] 3D multi-object tracking is a key component of autonomous driving systems because it provides crucial information to facilitate various onboard modules from perception and prediction to planning. LiDAR is the most commonly used sensor for autonomous vehicles to perceive their surroundings.

[0003] 3D multi-object tracking based on LiDAR points is one of the fundamental technologies in computer vision, and it has wide applications in various visual perception devices (such as surveillance cameras and LiDAR). LiDAR-based 3D multi-object tracking algorithms are also one of the key core technologies in the current field of autonomous driving.

[0004] Current 3D multi-object tracking (3D MOT) algorithms do not have a filtering and correction process for erroneous input observations. Based on erroneous observations, they will produce erroneous predictions, reducing prediction accuracy. Summary of the Invention

[0005] This application provides a tracking information processing method, program product, electronic device, and chip system. The method can improve the reliability of observations or fix problems of track disconnection and misconnection.

[0006] In a first aspect, embodiments of this application provide a tracking information processing method based on laser point clouds. The method includes: determining the confidence level corresponding to at least one first bounding box to be processed; the first bounding box is a bounding box pre-annotated to the tracking target based on a laser point cloud image; and filtering reliable inputs from at least one first bounding box based on the confidence level to obtain a reliable input sequence.

[0007] In one feasible implementation, determining the confidence level corresponding to at least one first bounding box to be processed includes: inputting the input data corresponding to at least one first bounding box to be processed into a prediction model, determining at least one second bounding box based on the output of the prediction model, and calculating the intersection-union ratio (IOU) between the first bounding box and the corresponding second bounding box as the confidence level corresponding to the first bounding box.

[0008] In one feasible implementation, filtering trusted inputs from at least one first bounding box based at least on confidence level includes adding bounding boxes with confidence levels greater than or equal to a first threshold into a trusted input sequence.

[0009] In one feasible implementation, the process of filtering trusted inputs from at least one first bounding box based at least on confidence level further includes: for a third bounding box in at least one first bounding box with a confidence level less than a first threshold, performing the following operations: calculating the iterative nearest neighbor (ICP) registration degree between the current frame and the previous or next frame corresponding to the third bounding box; and adding third bounding boxes with a confidence level less than the first threshold and greater than or equal to a second threshold, and an ICP registration degree greater than or equal to a third threshold, to the trusted input sequence.

[0010] In one feasible implementation, after calculating the iterative nearest neighbor (ICP) registration degree between the current frame and the previous and / or next frame corresponding to the third bounding box, the method further includes: determining the bounding boxes with a confidence level less than a second threshold and an ICP registration degree lower than a third threshold as bounding boxes to be corrected; correcting the bounding boxes to be corrected; and adding the corrected bounding boxes to the trusted input sequence.

[0011] In one feasible implementation, after obtaining the trusted input sequence, the method further includes: using a bounding box in the trusted input sequence as an anchor box; replacing bounding boxes of different sizes in the trusted input sequence with anchor boxes of the same size; determining virtual anchor points corresponding to each anchor box; and the virtual anchor point being the center point of the corresponding anchor box.

[0012] In one feasible implementation, after obtaining the reliable input sequence, the method further includes: determining N trajectories corresponding to the reliable input sequence, where N≥2; each trajectory includes at least one bounding box; sequentially inputting the m-th trajectory into the prediction model to obtain a first trajectory; the m-th trajectory is one of the N trajectories, 1≤m≤N; the first trajectory is the trajectory obtained by forward prediction of the m-th trajectory; inputting the first trajectory into the prediction model in reverse order to obtain a second trajectory; the second trajectory is the trajectory obtained by reverse prediction of the first trajectory; and filling in the missing bounding boxes in the m-th trajectory and / or the second trajectory based on the first trajectory and / or the second trajectory.

[0013] In one feasible implementation, after obtaining the second trajectory, the method further includes: calculating the matching degree between the second trajectory and N-1 trajectories; the N-1 trajectories are the trajectories other than the m-th trajectory among the N trajectories; if the matching degree between the n-th trajectory and the second trajectory meets a preset condition, determining that the n-th trajectory and the first trajectory are the same trajectory, and connecting the first trajectory and the n-th trajectory; wherein, n≠m, and 1≤n≤N.

[0014] In one feasible implementation, calculating the matching degree between the second trajectory and N-1 trajectories includes: generating a global temporal lookup table; the global temporal lookup table contains the number and timestamp corresponding to each bounding box; calculating the IOU value between the bounding boxes corresponding to the same timestamp in the second trajectory and N-1 trajectories in reverse chronological order according to the timestamps; and determining that the nth trajectory and the first trajectory are the same trajectory if the matching degree between the nth trajectory and the second trajectory meets a preset condition, including: determining that the nth trajectory and the mth trajectory are the same trajectory if the IOU value between x consecutive bounding boxes of the nth trajectory and the second trajectory is greater than a fourth threshold.

[0015] Secondly, embodiments of this application also provide a tracking information processing method based on laser point clouds. The method includes: determining N trajectories to be processed, where N≥2; each trajectory includes at least one bounding box; sequentially inputting the m-th trajectory into a prediction model to obtain a first trajectory; the m-th trajectory is one of the N trajectories, where 1≤m≤N; the first trajectory is the trajectory obtained by forward prediction of the m-th trajectory; inputting the first trajectory into the prediction model in reverse order to obtain a second trajectory; the second trajectory is the trajectory obtained by reverse prediction of the first trajectory; and filling in the missing bounding boxes in the m-th trajectory and / or the second trajectory based on the first trajectory and / or the second trajectory.

[0016] In one feasible implementation, after obtaining the second trajectory, the method further includes: calculating the matching degree between the second trajectory and N-1 trajectories in reverse chronological order; the N-1 trajectories are the remaining trajectories among the N trajectories excluding the m-th trajectory; if the matching degree between the n-th trajectory and the second trajectory meets a preset condition, determining that the n-th trajectory and the first trajectory are the same trajectory, and connecting the first trajectory and the n-th trajectory; wherein, n ≠ m, and 1 ≤ n ≤ N.

[0017] In one feasible implementation, calculating the matching degree between the second trajectory and N-1 trajectories includes: generating a global temporal lookup table; the global temporal lookup table contains the number and timestamp corresponding to each frame image; calculating the IOU value between the bounding boxes corresponding to the same timestamp in the second trajectory and N-1 trajectories in reverse chronological order according to the timestamps; and determining that the nth trajectory and the first trajectory are the same trajectory if the matching degree between the nth trajectory and the second trajectory meets a preset condition, including: determining that the nth trajectory and the mth trajectory are the same trajectory if the IOU value between x consecutive bounding boxes of the nth trajectory and the second trajectory is greater than a fourth threshold.

[0018] Thirdly, embodiments of this application also provide a tracking information processing method based on laser point clouds. The method includes: using one bounding box in at least one bounding box to be processed as an anchor box; replacing bounding boxes of different sizes in at least one bounding box with anchor boxes of the same size; determining virtual anchor points corresponding to each anchor box; and the virtual anchor point being the center point of the corresponding anchor box.

[0019] Fourthly, embodiments of this application also provide a computer program product, which includes a program that, when run by an electronic device, causes the electronic device to perform the method as described in any one of the first to third aspects above.

[0020] Fifthly, the present application also provides an electronic device for transmitting data via a data transmission interface. The electronic device includes: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of the first to third aspects above.

[0021] Sixthly, the present application also provides a storage medium including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method as described in any one of the first to third aspects above.

[0022] In a seventh aspect, the present application also provides a chip system, comprising: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the first to third aspects above. Attached Figure Description

[0023] Figure 1 It is a flowchart of related technologies;

[0024] Figure 2 This is a flowchart of the relevant technology.

[0025] Figure 3 This is an example diagram illustrating the application scenario of this application;

[0026] Figure 4 This is a schematic diagram showing the position of the method provided in the embodiments of this application within the overall processing flow architecture;

[0027] Figure 5 This is a schematic diagram of the system architecture of one embodiment of the method provided in this application in an autonomous driving scenario;

[0028] Figure 6This is a schematic diagram of the data processing flow in one embodiment of the method provided in this application;

[0029] Figure 7 This is a schematic diagram of the processing flow in one embodiment of the method provided in this application;

[0030] Figure 8 This is a schematic diagram of the anchor frame and virtual anchor points in one embodiment of the method provided in this application;

[0031] Figure 9 This is a schematic diagram comparing the virtual anchor sequence with the original input sequence in one embodiment of the method provided in this application;

[0032] Figure 10 This is a schematic diagram of bidirectional prediction in one embodiment of the method provided in this application;

[0033] Figure 11 This is a schematic diagram of reverse matching in one embodiment of the method provided in this application;

[0034] Figure 12 This is a schematic diagram of spatiotemporal consistency verification and trajectory matching in one embodiment of the method provided in this application;

[0035] Figure 13 This is a schematic diagram of the module corresponding to improvement point (1) in one embodiment of the method provided in this application;

[0036] Figure 14 This is a schematic diagram of the technical effect corresponding to the improvement point (1) in one embodiment of the method provided in this application;

[0037] Figure 15 This is a schematic diagram of the module corresponding to improvement point (3) in one embodiment of the method provided in this application;

[0038] Figure 16 This is a schematic diagram of the technical effect corresponding to the improvement point (3) in one embodiment of the method provided in this application;

[0039] Figure 17 This is a schematic diagram of the technical effect corresponding to the improvement point (3) in one embodiment of the method provided in this application;

[0040] Figure 18 This is a schematic diagram of the module corresponding to improvement point (2) in one embodiment of the method provided in this application;

[0041] Figure 19 This is a schematic diagram of three modules in one embodiment of the method provided in this application;

[0042] Figure 20This is a flowchart illustrating the implementation of the method provided in the application embodiment, which combines three modules.

[0043] Figure 21 These are example diagrams illustrating the technical effects achieved by the method provided in the application embodiments;

[0044] Figure 22 This is yet another example diagram illustrating the technical effect achieved by the method provided in the application embodiments;

[0045] Figure 23 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0048] Laser radar is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. Its working principle is to emit a detection signal (laser beam) towards the target, and then compare the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, relevant information about the target can be obtained, such as the target's distance, azimuth, altitude, speed, attitude, and even shape parameters, thereby enabling the detection, tracking, and identification of targets such as aircraft and missiles.

[0049] In the field of autonomous driving, LiDAR is a crucial sensor, and LiDAR-based perception is currently a hot research topic in vehicle autonomous driving. LiDAR obtains spatial perception of the surrounding environment by collecting spatial point cloud information. LiDAR point clouds can provide relatively accurate spatial location information, such as the spatial position, geometric dimensions, and orientation of objects. This perception information can be provided to autonomous driving perception algorithms for training and processing, thereby providing high-level decision-making input information for autonomous vehicles. It can also provide annotation information for vehicle autonomous driving perception data annotation platforms.

[0050] 3D multi-target tracking algorithms based on laser point clouds have wide applications in various visual perception devices, while 3D multi-target tracking algorithms based on LiDAR are one of the key core technologies in the current field of autonomous driving. To facilitate understanding of the improvements made to the solutions proposed in the embodiments of this application, two related technologies are listed below for illustration.

[0051] Related technology 1: such as Figure 1 As shown, among the 3D MOT-based solutions, one type of method is 3D multi-object tracking based on single-stage matching association. This related technology proposes evaluating tracking performance in 3D space and introduces two new evaluation metrics: AMOTA (Average Multiple Object Tracking Accuracy) and AMOTP (Average Multiple Object Tracking Precision), providing a unified quantitative evaluation standard for 3D MOT. Single-stage matching is fast while maintaining a certain level of accuracy, achieving a balance between speed and accuracy and meeting real-time requirements. However, the inventors discovered during their research that the single-stage matching association-based scheme also has the following drawbacks:

[0052] (1) There is no screening and correction process for incorrect input observations, which causes the incorrect input observations to have a continuous impact on the algorithm pipeline;

[0053] (2) Based on the trajectory mode of single frame, single frame, and single time sequence, there are many problems such as trajectory disconnection and misconnection, resulting in low quality;

[0054] (3) Single-stage matching has low recall and accuracy, and there is a significant bottleneck in tracking performance.

[0055] Related technology two: such as Figure 2 As shown, another type of method based on 3D MOT is 3D multi-object tracking based on two-stage matching association. This related technology is based on motion adaptive prediction and 3D loss matrix matching, with small cumulative error, high recall and precision. Moreover, based on two-stage matching, it can significantly improve the matching recall and accuracy, and to some extent overcome the precision bottleneck of single-stage matching. However, related technology two also has the following drawbacks:

[0056] (1) There is no screening and correction process for incorrect input observations, which will produce incorrect predictions based on incorrect observations;

[0057] (2) The trajectory mode based on single frame, single frame, and single time sequence has problems such as trajectory disconnection and misconnection;

[0058] (3) Matching and updating the model based on incorrect prediction results will cause incorrect input observations to have a continuous adverse effect on the algorithm pipeline.

[0059] In view of this, embodiments of this application propose a tracking information processing method based on laser point clouds to solve at least one of the above problems.

[0060] The method provided in this application can be applied to various application scenarios for detecting, tracking, or identifying targets based on lidar, such as autonomous driving, aerospace, and geographic exploration. For example, a typical application scenario of this method might be as follows: Figure 3 The autonomous driving scenario shown.

[0061] In autonomous driving applications, it is necessary to annotate laser point cloud images, such as... Figure 4 As shown, Figure 4 This is the main process for automated annotation of laser perception data for autonomous driving. LiDAR points are the data to be annotated. A 3D detection pre-labeling model is used to annotate the tracking targets in the LiDAR point cloud image, generating 3D bounding boxes for the tracking targets. A 3D multi-target tracking + tracking trajectory enhancement module further processes the pre-annotated 3D bounding boxes to improve trajectory quality, resulting in the final annotation result, which is a trajectory sequence with an ID number (track sequence ID_xx), where ID_xx represents the trajectory sequence numbered xx. A set of trajectory sequences can include at least one trajectory, and typically multiple trajectories.

[0062] The method proposed in this application can be implemented after obtaining 3D bounding boxes, and can be specifically applied to, for example... Figure 4 The 3D multi-target tracking + tracking trajectory enhancement module marked by the dashed box shown is an enhancement module, which is the product corresponding to the method provided in this application embodiment. This enhancement module can help automate the annotation of autonomous driving perception data and provide high-quality tracking trajectories to improve the efficiency of automated data annotation.

[0063] It should be noted that 3D multi-target tracking based on LiDAR point cloud input has wide applications in autonomous driving and automatic annotation. Currently used 3D multi-target tracking methods can all use the enhancement module proposed in the embodiments of this application. The enhancement module can be an application upgrade package or plugin, etc. The enhancement module can be stored in a removable storage medium and can be plugged in and used immediately when needed. Actual testing shows that the product obtained based on this method is efficient, performs well, and has wide application value.

[0064] Specifically, the products obtained based on the embodiments of this application can be software products such as functional plug-ins, upgrade packages, and applications, or electronic devices with the above-mentioned software products installed, such as laser sensing devices, vehicle monitoring devices, and lidar; or vehicles or automated labeling platforms for laser sensing data installed with the above-mentioned electronic devices.

[0065] For example, for vehicles equipped with laser sensing devices, the method provided in this application embodiment can be deployed on the computing nodes of these devices. Through software algorithms, the quality of multi-target tracking trajectories can be improved. Tracking trajectories, as an important source of information, can be used for autonomous driving perception decision-making, regulation and control, etc.

[0066] For example, for an automated labeling platform for laser sensing data, the method provided in this application embodiment can be deployed in an automated data labeling tool. Through the tool's algorithm, the quality of the tracking trajectory can be improved, providing a high-quality tracking trajectory, which can improve the efficiency and accuracy of automated labeling.

[0067] Specifically, the tracking information processing method based on laser point clouds provided in this application has made improvements in at least the following three aspects: (1) performing confidence and / or ICP registration calculations on the original input; (2) proposing the use of virtual anchor points; and (3) trajectory matching based on reverse prediction. The above three improvements can be implemented individually or in combination, and thus multiple embodiments can be obtained, which are described below.

[0068] Based on aspect (1), in some embodiments, the tracking information processing method based on laser point clouds provided in this application performs at least a confidence estimate on the original input to filter out erroneous input information. Erroneous input information can interfere with and mislead the motion prediction model. Therefore, motion prediction based on erroneous input information often results in erroneous prediction results. The method proposed in this application can reduce erroneous information from the input, reduce erroneous prediction results, and improve the overall performance of the algorithm.

[0069] like Figure 5 As shown, Figure 5 This diagram illustrates the application of the method provided in this embodiment of the application in an autonomous driving scenario. The method provided in this embodiment first predicts the original observations to obtain a pred box (the predicted bounding box, i.e., the second bounding box). The Intersection over Union (IOU) value between the pred box and the original 3D box is calculated as the confidence score of the original observation 3D box. Based on the confidence score, it is determined whether the 3D box is a reliable input, thus obtaining a reliable input sequence. The 3D boxes in the reliable input sequence are those with a high degree of overlap with the pred box.

[0070] Specifically, the method may include the following operations:

[0071] S10, determine the confidence level corresponding to at least one first bounding box to be processed.

[0072] In this embodiment, the first bounding box is a bounding box pre-annotated based on the laser point cloud image of the tracking target. Generally, the first bounding box is a bounding box obtained by detecting a pre-annotated model in 3D. For example, Figure 4 The 3D bounding boxes shown are the first bounding boxes. The first bounding box can be understood as the original input to be processed or the uncertain observation.

[0073] In some embodiments, such as Figure 6 As shown, the confidence level corresponding to at least one first bounding box to be processed can be determined in the following way:

[0074] The input data corresponding to at least one first bounding box to be processed is input into the prediction model. Based on the output of the prediction model, at least one second bounding box is determined. Then, the intersection-union ratio (IOU) between the first bounding box and the corresponding second bounding box is calculated as the confidence level corresponding to the first bounding box.

[0075] That is, the motion trajectory of the first bounding box is predicted by the prediction model, and the bounding box corresponding to the motion trajectory of the first bounding box at the next time moment is output, which is the second bounding box.

[0076] Optionally, the prediction model can be a Kalman filter. For example, in some embodiments, a lossless Kalman filter can be used, while in other embodiments, an extended Kalman filter or a standard Kalman filter can be used. In other embodiments, it is not excluded that other prediction models with motion trajectory prediction capabilities can be used instead of the Kalman filter, such as prediction models based on LSTM networks to predict the future motion trajectory of moving targets.

[0077] Intersection Over Union (IOU) measures the degree of overlap between two bounding boxes and is commonly used as an evaluation metric for object detection tasks. For example, the specific steps for calculating IOU are as follows: Calculate the area of ​​the intersection region of the two bounding boxes (i.e., the intersection area); calculate the area of ​​the union region of the two bounding boxes (i.e., the union area); and obtain the IOU value by dividing the intersection area by the union area. The calculation formula is as follows: IOU = Intersection Area / Union Area.

[0078] S11, based at least on confidence level, filter trusted inputs from at least one first bounding box to obtain a trusted input sequence.

[0079] Figure 6 The original input sequence can include multiple first bounding boxes. After obtaining the confidence scores of the first bounding boxes, it can be determined whether the current first bounding box is a reliable input, at least based on the confidence scores.

[0080] In some embodiments, trusted inputs can be selected from the original input sequence based solely on confidence levels. For example, bounding boxes with a confidence level greater than or equal to a first threshold are added to the trusted input sequence. Specifically, a first threshold can be preset, which is a confidence threshold. If the confidence level is greater than or equal to the first threshold, the first bounding box is considered a trusted input and added to the trusted input sequence. This process continues until the corresponding IOU value of each first bounding box in the original input sequence is calculated and compared with the first threshold, thus obtaining the trusted input sequence.

[0081] The first threshold can have various values, such as 0.9, 0.8, 0.95, or 0.85, 0.86, 0.92, 0.75, 0.78, etc. For example, a first bounding box with a confidence level greater than or equal to 0.9 is considered a reliable input and added to the reliable input sequence, while a first bounding box with a confidence level less than 0.9 is temporarily judged as an unreliable input. The specific value of the first threshold can be set by those skilled in the art according to the actual application scenario, and this specification does not list them all.

[0082] In other embodiments, after performing a first-level screening using confidence scores, trusted inputs are further filtered out by calculating ICP registration scores. Optionally, in some embodiments, bounding boxes that are not selected as trusted inputs can be corrected before being added to the trusted input sequence.

[0083] Specifically, for bounding boxes in the original input sequence with a confidence level less than the first threshold (defined as third bounding boxes to prevent confusion), the following operations are performed:

[0084] Calculate the iterative nearest neighbor (ICP) registration degree between the current frame and the previous or next frame corresponding to the third bounding box; add the third bounding boxes with confidence scores less than the first threshold and greater than or equal to the second threshold, and ICP registration degrees greater than or equal to the third threshold, to the trusted input sequence.

[0085] Among them, ICP (Iterative Closest Point) is essentially an optimal registration method based on the least squares method. ICP is a commonly used point cloud registration method. Its basic idea is to align corresponding points between two point clouds iteratively, thereby achieving point cloud registration. For example, the specific steps of the ICP algorithm can be as follows:

[0086] Corresponding point matching: For each point in the point cloud to be registered, find its nearest neighbor in the reference point cloud and establish a correspondence;

[0087] Calculate the transformation matrix: Based on the corresponding point pairs, calculate the transformation matrix of the point cloud to be registered relative to the reference point cloud. For example, the least squares method or SVD decomposition method can be used for the transformation.

[0088] Applying the transformation matrix: Applying the transformation matrix to the point cloud to be registered yields a new point cloud;

[0089] Convergence check: Determine if the transformation matrix satisfies the convergence condition. If it does, stop the iteration.

[0090] In other embodiments, ICP registration between different frames can be combined with other algorithms to improve registration accuracy and robustness.

[0091] In some embodiments, the ICP registration degree between the current frame and the previous frame can be calculated as the ICP registration degree corresponding to the current frame. In another embodiment, the ICP registration degree between the current frame and the next frame can be calculated. In still other embodiments, both the ICP registration degree between the current frame and the previous frame and the ICP registration degree between the current frame and the next frame can be calculated, and the average of the ICP registration degrees calculated for the previous and next frames can be used as the ICP registration degree corresponding to the current frame.

[0092] After calculating the ICP registration between the current frame and the previous or next frame corresponding to the third bounding box, the following operations can also be performed:

[0093] Bounding boxes with confidence levels below the second threshold and ICP registration levels below the third threshold are identified as bounding boxes to be corrected; these bounding boxes are corrected; and the corrected bounding boxes are added to the trusted input sequence.

[0094] The purpose of calculating ICP is to further filter out trustworthy inputs from the sequence of bounding boxes temporarily judged as untrustworthy inputs. The second threshold is also a confidence threshold, but it is lower than the first threshold, and the third threshold is the ICP threshold. For example, the second threshold can be 0.5, 0.6, 0.65, 0.55, 0.63, or 0.52, etc.; the third threshold can be 0.7, 0.75, 0.8, 0.72, or 0.69, etc. For instance, if the second threshold is 0.5 and the third threshold is 0.7, then in the sequence of bounding boxes temporarily judged as untrustworthy inputs with a confidence level less than 0.9, for bounding boxes with a confidence level greater than or equal to 0.5, their ICP value is further checked to see if it is greater than 0.7. If both the confidence level and the ICP value are greater than 0.7, then the bounding box is determined to be a trustworthy input and added to the trustworthy input sequence.

[0095] In some embodiments, bounding boxes with a confidence level less than 0.5 and an ICP value less than 0.7 are considered untrusted inputs and require correction before being added to the correction sequence. In other embodiments, correction may be omitted, and bounding boxes with a confidence level less than 0.5 and an ICP value less than 0.7 are simply discarded and not used.

[0096] In some embodiments, bounding boxes that do not meet the conditions of confidence level less than 0.5 and ICP value less than 0.7, nor do they meet the conditions of confidence level greater than or equal to 0.5 and ICP value greater than or equal to 0.7, are also included in the sequence to be corrected.

[0097] In some embodiments, the bounding box can be modified as follows:

[0098] Determine the center coordinates (x, y, z) of the bounding box. Adjust at least the x and y values ​​according to a preset step size, for example, x + a, y + a (where a is the step size). Calculate the ratio of the number of laser points within the bounding box to the total number of laser points in the entire frame. If the ratio increases, the adjustment direction is correct, and you can continue to increase the step size 'a' in x and y. If the ratio decreases after adjustment (meaning fewer points fall within the bounding box), the adjustment direction is incorrect and needs to be reversed. The correction ends when the proportion of laser points within the bounding box in the global point cloud is maximized. This achieves correction for untrusted inputs. The corrected bounding box can then be added to the trusted input sequence. It should be noted that in autonomous driving scenarios, adjusting the z value is generally unnecessary, but in some embodiments, it may be possible.

[0099] The following is a specific example, such as Figure 7 As shown, this specific embodiment may include the following process:

[0100] S710 inputs the original input (first bounding box) into the prediction model and outputs the second bounding box.

[0101] The original input is a 3D bounding box with attribute values ​​(x, y, z, w, h, l, θ, s). Here, x, y, and z represent the coordinates of the center point of the bounding box.

[0102] Initial motion prediction is performed using a prediction model, that is, predicting the motion trajectory of the first bounding box at the next moment, to obtain the second bounding box.

[0103] S711, perform confidence estimation on the original input of the current frame.

[0104] Specifically, the spatial overlap value (IOU value) between the second bounding box and the original input (first bounding box) is calculated, and the overlap value is the confidence level of the original input (first bounding box) in the current frame.

[0105] S712, compare the confidence level obtained in S711 with the first threshold, and add the original inputs that are greater than the first threshold to the trusted input sequence.

[0106] S713, the original inputs that are less than the first threshold are then ICP registered with the global point cloud.

[0107] Global point cloud refers to all the point clouds in a single frame of laser point cloud image corresponding to the bounding box. In other words, the point cloud in the entire image is the global point cloud, while the point cloud within the bounding box or detection box is the local point cloud.

[0108] ICP registration is performed between the original input and the global point cloud. That is, ICP registration is performed between the current frame of the laser point cloud image containing a bounding box in the original input sequence and the previous or next frame of the laser point cloud image.

[0109] S714. Bounding boxes with a confidence level greater than or equal to the second threshold and an ICP value greater than or equal to the third threshold are added to the trusted input sequence, while other bounding boxes are added to the sequence to be corrected.

[0110] S715, and correct the bounding boxes in the sequence to be corrected (i.e., observation correction), and then add the corrected observations to the reliable input sequence.

[0111] For example, the correction method could be: calculate the proportion of the number of points in the bounding box before correction to the number of points in the global point cloud in the current frame image, then make corrections, with the center point coordinates of the bounding box being x+1 and y+1, and then calculate the proportion of the local point cloud to the global point cloud. If the proportion increases, continue to add 1 and calculate again; if the proportion decreases, try to calculate x-1 and y-1, and so on, until the proportion is maximized.

[0112] Matching and updating the model based on erroneous prediction results will carry the erroneous information into the entire running cycle of the motion model, causing irreversible errors in the prediction of future frames. The above embodiments of this application can solve the problem of continuous errors caused by matching and updating the motion model based on erroneous prediction results, improve the reliability of the input, reduce the impact of erroneous information on the subsequent processing flow, provide more accurate observation data, and thus improve the prediction accuracy.

[0113] Based on the improvement in aspect (2), in some embodiments, the tracking information processing method based on laser point clouds provided in this application proposes to use anchor boxes and virtual anchor points to replace the bounding boxes and center points in the original input. See [reference needed]. Figure 8 and Figure 9As shown, the specific operations may include the following:

[0114] S20, using one bounding box of at least one bounding box to be processed as the anchor box;

[0115] In step S20, the at least one bounding box to be processed can be at least one bounding box in the trusted input sequence. In other words, the processing object in S20 can be the trusted input sequence obtained in the above embodiments. In some embodiments, the first bounding box in the trusted input sequence can be used as the anchor box. In other embodiments, to avoid errors caused by inaccurate detection of the first bounding box, a bounding box can be selected from the trusted input sequence as the anchor box, not limited to the first bounding box. For example, the (t+a)th bounding box in the trusted input sequence can be used as the anchor box.

[0116] It should be noted that the processing object of S20 is not limited to the trusted input sequence obtained in the above embodiments, but can also be the original input, i.e. Figure 4 The 3D Bounding boxes shown.

[0117] S21, replace the bounding boxes of different sizes in at least one bounding box with anchor boxes of the same size.

[0118] like Figure 8 As shown, in the laser point cloud image of frame t, the original bounding boxes are significantly larger than those in the previous frame ta and the subsequent frame t+a. However, in practical applications, most tracked targets do not undergo significant size changes within a tracking time series. Figure 8 The scenario shown might involve a large error in the pre-labeled bounding box of frame t. If this error is not corrected, it could affect the accuracy of subsequent processing. Therefore, in this embodiment, a pre-labeled bounding box (i.e., a pre-labeled bounding box, specifically a bounding box in a trusted input sequence) is selected from the sequence to be processed as the anchor box, and then all bounding boxes in the sequence are replaced with the anchor box.

[0119] Specific replacement methods can be as follows: Figure 8 As shown, taking frame t as an example, the left edge of the anchor box is aligned with the left edge of the original pre-labeled box (in chronological order), and the anchor box replaces the original pre-labeled box for subsequent matching, association and other calculations.

[0120] S22, determine the virtual anchor point corresponding to each anchor box, where the virtual anchor point is the center point of the corresponding anchor box.

[0121] After determining the anchor boxes, calculate the center point of each anchor box. The center point of each anchor box is the virtual anchor point, which replaces the center point of the original pre-defined anchor box. For example, ... Figure 8As shown, in frame t, the solid black dots are virtual anchor points, and the hollow elliptical dots are the center points of the original pre-labeled boxes. The virtual anchor points replace the center points of the original pre-labeled boxes. In other words, in subsequent processing and calculations, in each frame of the image, the anchor box is used as the bounding box, and the virtual anchor point is used as the center point in the calculation.

[0122] One possible implementation is that the virtual anchor point can be the geometric center point of the anchor frame.

[0123] Figure 8 The ground truth (GT) boxes in the image represent the true boundaries of the tracked object. The center point of the GT box is indicated by a dashed circle. The purpose of showing the GT boxes is to demonstrate that the anchor boxes are closer to the ground truth relative to the original pre-labeled boxes.

[0124] like Figure 9 As shown, after processing, the bounding boxes in the input sequence (e.g., a trusted input sequence) are all of the same size in the resulting anchor box sequence. The virtual center point (virtual anchor point) does not change with the size of the box, thus reducing the impact of layout errors on the accuracy of subsequent calculations.

[0125] Another problem that needs to be overcome in the 3D MOT algorithm is the disconnection and misconnection of tracking trajectories, resulting in identity switch (IDs) issues. This seriously affects the accuracy of autonomous vehicle control and decision-making, and also has a significant impact on annotation efficiency in automated annotation of moving targets.

[0126] How to reduce and correct disconnections and misconnections in tracking trajectories and provide complete, high-quality trajectories is a research challenge and pain point in 3D MOT. Solving this problem has a clear business need in the annotation of moving targets in autonomous driving perception data and has significant commercial value in improving automatic annotation efficiency and reducing costs.

[0127] In view of this, in some embodiments, the tracking information processing method based on laser point clouds provided in this application also proposes a trajectory correction module for spatiotemporal consistency verification and bidirectional prediction. The module performs consistency verification on the sequence-level tracking trajectory and performs prediction and box filling on the lost frame sequence through bidirectional (sequential and reverse) prediction. This is used to solve the problem of trajectory disconnection caused by pre-labeled frame loss and missed detection. The consistency of the trajectory is improved through temporal consistency verification and the integrity of the trajectory is improved through bidirectional prediction.

[0128] This method may specifically include the following operations:

[0129] S30, determine the N trajectories to be processed.

[0130] Where N is an integer, and N≥2. Determining the N trajectories to be processed means determining how many trajectories are identified in the global temporal frame images to be processed. Each trajectory corresponds to at least one frame of laser point cloud image. Each laser point cloud image may or may not contain bounding boxes; that is, the boundary of the target may not be identified in some images. A trajectory may include at least one bounding box. A global temporal frame image is a segment of continuous frames of laser point cloud images acquired by the lidar to be processed. Full temporal sequence means the frames are continuous and temporally complete.

[0131] It should be noted that in the embodiments of this application, a set of input sequences may contain multiple bounding boxes in a full-time image frame, and multiple bounding boxes may constitute multiple trajectories. Therefore, the N trajectories to be processed can be the N trajectories corresponding to the input sequence. The input sequence can be a reliable input sequence, a reliable input sequence after replacing the center point with a virtual anchor point, or the original input sequence.

[0132] S31. Input the m-th trajectory sequentially into the prediction model to obtain the first trajectory.

[0133] The m-th trajectory is one of the N trajectories, where 1 ≤ m ≤ N. For example... Figure 10 As shown, the first trajectory is the trajectory obtained by forward prediction of the m-th trajectory. For example, if m = 1, then the m-th trajectory is the 1st trajectory, and the first trajectory is the trajectory obtained by forward prediction of the 1st trajectory in the sequence to be processed.

[0134] In this application embodiment, the sorting according to chronological order is called sequential, and the sorting in the reverse order of chronological order is called reverse order. For example, such as Figure 10 As shown, the m-th trajectory corresponds to frame 1 to frame t+1, so the sequential input sequence is: frame 1, frame 2, ..., frame t, frame t+1.

[0135] S33, input the first trajectory in reverse order into the prediction model to obtain the second trajectory.

[0136] The second trajectory is obtained by reversing the prediction of the first trajectory. After obtaining the first trajectory, the first trajectory is input into the prediction model in reverse order. The reverse sequence corresponding to the first trajectory is: frame t+2, frame t+1, frame t, ..., frame 2, frame 1.

[0137] S34, based on the first trajectory and / or the second trajectory, fill in the missing bounding boxes in the m-th trajectory and / or the second trajectory.

[0138] Trajectories obtained from laser point clouds may contain some missed boxes, causing the trajectory to break. During the research process, the inventors discovered that by first performing forward prediction on the same trajectory and then performing reverse prediction, some missed boxes can be predicted. Thus, the missed boxes can be filled in through bidirectional prediction.

[0139] like Figure 12 As shown, the box marked 'e' is the missed box. The missing gaps can be filled in by the second trajectory obtained through reverse prediction. For example, the box marked 'b' is the filled box.

[0140] In step S34, the missing bounding boxes in the m-th trajectory and / or the second trajectory are filled in, including filling in only the m-th trajectory, only the second trajectory, or both the m-th trajectory and the second trajectory. In most embodiments, at least the second trajectory is filled in.

[0141] This application embodiment also connects the disconnected trajectories through spatiotemporal consistency verification, and the specific implementation can be as follows:

[0142] After obtaining the second trajectory, the matching degree between the second trajectory and N-1 trajectories is calculated in reverse chronological order. The N-1 trajectories are all trajectories except the m-th trajectory among the N trajectories. The reverse chronological order can be based on the timestamps of the multiple laser point cloud image frames corresponding to the second trajectory, in reverse chronological order.

[0143] Specifically, as one possible implementation method, the matching degree between the second trajectory and N-1 trajectories can be calculated in the following way:

[0144] The laser point cloud images corresponding to the N trajectories are numbered according to the timestamp of each frame to generate a global temporal lookup table. The global temporal lookup table contains the number and timestamp of each frame image. The IOU value between the bounding boxes corresponding to the same timestamp in the second trajectory and the N-1 trajectories is calculated in reverse order of timestamp.

[0145] For example, such as Figure 10 and Figure 11 As shown, the second trajectory corresponds to frames 3 to t+3, so the reverse matching order is: frame t+3, frame t+2... frame 4, frame 3.

[0146] Specifically, such as Figure 11 As shown, assuming m=1, the matching degree between the second trajectory and N-1 trajectories is calculated in reverse time order. That is, the matching of the second trajectory with other trajectories (trajectory 02 to trajectory N) is performed first in frame t+3, then the matching of the second trajectory with other trajectories is performed in frame t+2, and so on.

[0147] If the matching degree between the nth trajectory and the second trajectory meets a preset condition, the nth trajectory is determined to be the same trajectory as the first trajectory, and the first trajectory and the nth trajectory are connected, where n ≠ m and 1 ≤ n ≤ N. The nth trajectory is a trajectory among the N trajectories that is different from the mth trajectory.

[0148] For example, the matching degree between the nth trajectory and the second trajectory meets the preset condition, which can be that the IOU value between x consecutive bounding boxes of the nth trajectory and the second trajectory is greater than the fourth threshold, where x is an integer and can take values ​​of 2 / 3 / 4 / 5, etc. The fourth threshold can be 0.8 / 0.82 / 0.85 / 0.76 / 0.78, etc. For example, if x=3, and the IOU value between 3 consecutive bounding boxes of the nth trajectory and the second trajectory is greater than 0.8, then the nth trajectory and the mth trajectory are considered to be the same trajectory.

[0149] The following is a specific example, such as Figure 12 As shown, in this specific embodiment, the scene's full-time frames include frame T1, frame T2, ..., frame Tn-2, frame Tn-1, and frame Tn. The first trajectory is the trajectory obtained through forward prediction, and the second trajectory is the direction-predicted trajectory obtained by inputting the first trajectory in reverse order. The specific implementation steps are as follows:

[0150] The complete scene is encoded in full time sequence to generate a global time sequence lookup table.

[0151] In the global temporal sequence, based on the pre-labeled temporal segments, the first trajectory (trajectory 01) in the sequence is predicted by forward inference according to the sequential temporal order to obtain the first trajectory 1; that is, in this embodiment, m=1. It should be noted that the first trajectory and the first trajectory should not be confused, and the first trajectory is the forward prediction result of the first trajectory.

[0152] In the global time series, based on the first trajectory, the second trajectory is obtained by reasoning backward from the tail node of the first trajectory (in reverse order).

[0153] The spatiotemporal consistency of the second trajectory with N-1 trajectories (trajectory 02 to trajectory N) is checked. By using the sequence-level trajectory status and a global temporal lookup table, the trajectories are aligned on the timestamp. The aligned trajectory boxes are found by temporal lookup, and the matching IOU value is calculated in the spatial dimension to complete the spatiotemporal consistency check.

[0154] Trajectory correction aligns the forward predicted trajectory with the reverse predicted trajectory. Based on the spatial dimension matching IOU value, it can correct disconnected or misconnected trajectories. When the IOU value between trajectory n and the second trajectory is greater than a certain threshold (e.g., 0.8), trajectory 01 and trajectory n are connected.

[0155] The method proposed in this application can at least solve the following problems: In trajectory modes based on single frames, single frames, and single temporal directions, there are issues with disconnected or incorrectly connected trajectories. Single-frame, single-frame trajectories typically have poor correlation between trajectories. When the algorithm encounters more complex scenarios, trajectories may become disconnected or incorrectly connected. Furthermore, unidirectional prediction makes it difficult to recover trajectories for targets that have been occluded for extended periods. The method proposed in this application can complete disconnected trajectories through bidirectional prediction and perform direction matching calculations between the predicted trajectories and other trajectories to identify two trajectories that are actually the same trajectory but have been identified as different trajectories and thus have different IDs. Connecting these two trajectories is equivalent to intelligently connecting disconnected trajectories using the aforementioned algorithm, thus compensating for the shortcomings of hardware detection.

[0156] Based on the above description, it can be seen that the embodiments of this application propose at least three improvements: (1) by using confidence or combining ICP registration, the credibility of the original input is judged, and untrustworthy erroneous input information is filtered out to improve the reliability of the input; (2) the technical concept of virtual anchor points is proposed, and the center point of the original annotation box is replaced by virtual anchor points; (3) a trajectory bidirectional prediction and reverse matching mechanism is proposed to fill the missed boxes and connect the disconnected same trajectory.

[0157] It should be noted that the above improvements can be implemented individually or in combination. Individual implementation can yield multiple embodiments, while combined implementation can yield even more embodiments. The above descriptions are mostly from the perspective of individual implementations; the following descriptions will focus on the perspective of combined implementations.

[0158] like Figure 13 As shown, in a combined embodiment, the module to which the improvement (1) proposed in this application belongs in the entire solution is as follows: Figure 13 The solid line marks the module shown. In this embodiment, confidence is estimated for observations (original input) with uncertainty. Based on the confidence value and the auxiliary judgment of global point cloud information of previous and subsequent frames, erroneous observations are filtered out; erroneous observations are corrected; and matching association and motion model updates are performed based on reliable observations.

[0159] The specific implementation steps can be as follows:

[0160] Initial motion prediction based on uncertain observations.

[0161] Based on the current frame, iterate through the global point cloud information of the previous and next input frames.

[0162] Confidence estimation involves calculating the Interchange of Union (IOU) between the original input bounding boxes (3D Object Detections) and the predicted bounding boxes (Tracks), which is used as the confidence value of the original input. Inputs with a confidence value greater than or equal to a first threshold are directly added to the trusted input sequence (Dets). For original inputs with a confidence value less than the first threshold, the following steps are performed:

[0163] Based on the confidence score and the ICP registration accuracy of the global point cloud information in the preceding and following frames, an auxiliary judgment is made. Input boxes (bounding boxes) with a confidence score less than the first threshold but greater than or equal to the second threshold, and an ICP registration accuracy greater than the third threshold, are marked as reliable observations and added to Dets. Input boxes with a confidence score less than the second threshold and an ICP registration accuracy less than the third threshold are filtered out, marked as observations to be corrected, and their size is corrected. The corrected boxes are then added to the reliable observations. The specific correction method is described above and will not be repeated here.

[0164] Once reliable observations are obtained, subsequent correlation matching and motion model updates are performed based on these reliable observations.

[0165] Therefore, this application proposes an observation confidence estimation and correction module to estimate the confidence of the input observations and correct erroneous input values ​​(observations), thereby solving the problem of the impact of erroneous input information on model performance, realizing reliable observation input, and improving the prediction accuracy and performance of the tracking model.

[0166] The beneficial effects of the various embodiments obtained based on the improvement (1) proposed in this application are as follows: Figure 14 As shown, Figure 14 The image shows the 2D effect from the BEV (Bird's Eye View) perspective. The white box is the observation box before optimization, and the gray box is the observation box after optimization. After optimization, the spatial position accuracy of the observation box is higher, the error is smaller, and the registration with the original laser point cloud is higher.

[0167] This application also provides a trajectory correction module based on spatiotemporal consistency verification and bidirectional prediction. The modules in the overall scheme of the various embodiments derived from this module are as follows: Figure 15 As shown (the modules marked with black solid lines in the figure). In this embodiment, based on the consistency of the time and spatial dimensions, consistency verification is performed on the sequence-level tracking trajectory. The complete full-time sequence frames are encoded. For the sequence-level trajectory to be corrected, consistency alignment and search are performed in the time dimension to find the trajectory missing frame sequence. Through bidirectional (sequential and reverse) prediction, the missing frame sequence is predicted and filled in. At the same time, consistency verification is performed in the spatial dimension to reconnect the broken trajectory.

[0168] Figure 15In this context, 3D object detections can represent either the original input (uncertain observations) or observations from a reliable input sequence (first bounding boxes). Kalman Filter model represents the Kalman filter model, Tracks represent the trajectories predicted by the Kalman filter model (e.g., first or second trajectory), Virtual Anchor Association represents virtual anchor association, Unmatched Dets represent unmatched inputs (bounding boxes), Unmatched Tracks represent unmatched tracks, Matched Dets / Tracks represent matched inputs or tracks, and Lifemanagemer (delete / remain) represents the life manager, which manages the validity of the original inputs (observations).

[0169] The beneficial effects of the multiple embodiments obtained based on the above-mentioned improvement point (3) are as follows: Figure 16 As shown, in a tracking scenario, before optimization, the tracking trajectory was broken due to missed pre-labeling. After optimization, through spatiotemporal consistency verification and bidirectional prediction, reverse box filling is performed at the missed frame. Embodiment 2 of the present invention can reconnect the broken trajectory, repair the tracking ID, and significantly reduce ID jump indicators.

[0170] like Figure 17 As shown, side ① represents an example of recognition results in related technologies; side ② represents an example of recognition results in the embodiments of this application. By employing the trajectory correction module based on spatiotemporal consistency verification and bidirectional prediction proposed in the embodiments of this application, consistency verification is performed on the sequence-level tracking trajectory. Through bidirectional (sequential and reverse) prediction, predicted bounding boxes are filled in for the lost frame sequence, resulting in recognition results with richer details. It has a good effect on pain points such as trajectory disconnection caused by pre-labeled frame loss and missed detection. The temporal consistency verification improves the consistency of the trajectory, and the bidirectional prediction improves the integrity of the trajectory.

[0171] This application also provides a matching and updating module based on virtual anchor points. The modules belonging to the overall solution in multiple embodiments obtained based on this module are as follows: Figure 18 As shown in the figure (the module is marked by a solid black line). In these embodiments, the target tracking center is adjusted from the center of the pre-labeled bounding box to a normalized virtual center position that is unaffected by the category, avoiding center point errors caused by changes in the size of the pre-labeled bounding box. Matching and updating based on this virtual anchor point can improve the accuracy and robustness of the algorithm.

[0172] For example, the specific implementation steps can be as follows:

[0173] Use the first frame annotation box of the trajectory as the anchor box;

[0174] Based on the anchor box, the virtual anchor point is calculated in a normalized manner. According to the size of the anchor box, a virtual center point is refitted for the point cloud position center of the current frame box to replace the actual center of the current box. This virtual center point (virtual anchor point) will not change with the size of the box.

[0175] Matching is based on virtual anchor points;

[0176] Based on the matching results, the motion model (i.e., the prediction model) is updated.

[0177] Therefore, this application proposes a matching and updating module based on virtual anchor points, which adjusts the target tracking center from the center of the pre-annotated box to a normalized virtual center position that is unaffected by the category, in order to solve the problem of changes in the size of the pre-annotated box (which may be caused by size changes or category misdetection), and improve the continuity and consistency of the trajectory.

[0178] In summary, this application proposes at least one of the following modules: an observation confidence estimation and correction module, a trajectory correction module based on spatiotemporal consistency and bidirectional prediction, and a matching and updating module based on virtual anchor points. Multiple embodiments can be obtained by using one or more of the above three modules. For example, in one embodiment, the above three modules are used in combination, and the positions of the three modules in the overall flowchart are as follows: Figure 19 As shown (represented by solid black lines).

[0179] The following is a specific example of the combined use of the above three modules.

[0180] like Figure 20 As shown, this specific embodiment may include the following process:

[0181] S2010: The original input (first bounding box, i.e., 3D bounding box) is passed through a prediction model (e.g., Kalman filter) to obtain the second bounding box.

[0182] Initial motion prediction is performed using a prediction model, that is, predicting the motion trajectory of the first bounding box at the next moment, to obtain the second bounding box.

[0183] S2011, perform confidence estimation on the current frame.

[0184] Specifically, the spatial overlap (IOU) value between the second bounding box and the first bounding box is calculated. The IOU value is the confidence level of the first bounding box in the current frame.

[0185] S2012, compare the confidence level obtained in S2011 with a first threshold (e.g., 0.9), and add the first bounding box that is greater than the first threshold to the trusted input sequence.

[0186] The first threshold is the confidence threshold, which can be preset.

[0187] S2013, the original inputs that are less than the first threshold (e.g., 0.9) are then ICP registered with the global temporal point cloud.

[0188] For example, ICP registration is performed between the current frame and the previous frame to obtain the ICP value corresponding to the current frame. Figure 20 The global point cloud series attributes shown are (p1, p2, ..., p z ,…,p Z ), where Z represents the total number of points in the global point cloud of the current frame image, i.e., the total number of points contained in the laser point cloud of the current frame; z represents the z-th laser point; p1, p2, ..., p z These represent the spatial coordinates (i.e., three-dimensional coordinates) of the 1st, 2nd, and zth points in the laser point cloud of the current frame image, respectively, where 1 ≤ z ≤ Z.

[0189] S2014, determine whether the confidence level of the current frame is greater than or equal to the second threshold 0 (e.g., 0.5) and whether it is greater than or equal to the third threshold (e.g., 0.7). If the condition is met, the bounding box is considered a reliable input and is added to the reliable input sequence.

[0190] Add the remaining bounding boxes to the sequence to be corrected.

[0191] The second threshold is the confidence threshold, and the third threshold is the ICP threshold. The second and third thresholds can be preset.

[0192] S2015, correct the attributes of the input in the sequence to be corrected (observation correction).

[0193] The corrected observations (i.e., bounding boxes) are then added to the trusted input sequence. The correction of the bounding boxes can involve modifying at least one of their attribute values. Specifically, the attributes of the bounding boxes can include (x, y, z, w, h, l, θ, s). Here, x, y, and z represent the coordinates of the bounding box's center point; w, h, and l represent the width, height, and length of the bounding box, respectively; θ represents the yaw angle; and s represents the confidence score. For example, the confidence score can be a numerical value obtained by evaluating the confidence of the bounding box using a 3D detection pre-calibration model. It should be noted that the inventors discovered in their research that the s value is not necessarily accurate and may contain a certain degree of error. Therefore, this application proposes a new confidence evaluation mechanism.

[0194] Furthermore, in some embodiments, the bounding box correction may only involve correcting the coordinates of the center point (e.g., adjusting the x and y coordinate values); in other embodiments, the center point coordinates and at least one of the bounding box's width w, height h, and length l may be corrected. The correction of width, height, and length can refer to the coordinate adjustment process described above, and will not be repeated here. Generally, the yaw angle θ and confidence score s are not corrected, but it is possible in some embodiments that the yaw angle θ and confidence score s may be corrected.

[0195] S2016, Calculate the virtual anchor point: For a reliable input sequence, take the first frame box of the sequence as the anchor box, and calculate a virtual anchor point based on the anchor box. The virtual anchor point replaces the actual center of all boxes in the sequence.

[0196] S2017, matching and association based on virtual anchor points:

[0197] Bounds larger than the matching threshold (fourth threshold) are marked as matched and a tracking ID is output. Matched boxes (trajectory / boundary box) continue to be used for motion model updates. Bounds smaller than the matching threshold are marked as unmatched and added to the trajectory management cycle sequence. Bounds in the trajectory management cycle sequence are marked as discarded after meeting certain conditions.

[0198] Specifically, for unmatched boxes and boxes that meet temporal consistency requirements, bidirectional prediction and rematching are performed, and the final trajectory is output. Steps S2010-S2017 are repeated until tracking and trajectory output for all frames are completed.

[0199] The method proposed in the embodiments of this application was used to conduct evaluations on a real business test set.

[0200] The beneficial effects of the method provided in this application embodiment were verified by selecting the real business set Test_City. Test_City contains 9 complex urban road scenarios, with categories including car, truck, ped, and bicycle. The ground truth (GT) of each scenario is labeled by two annotators. During testing, the GT is divided into two segments (as shown in Table 1, segment gt_1 and segment gt_2).

[0201] Table 1

[0202]

[0203]

[0204] It can be seen that the solution provided in this application significantly improves performance compared to the three related technologies. Further combining... Figure 21 , Figure 22As shown, the following beneficial effects can be achieved:

[0205] In nine segmented test sets within the urban area, the solution provided in this application significantly reduces the number of trajectory breaks compared to some related technologies, and also demonstrates superior performance in actual business implementation.

[0206] Compared with related self-developed solutions, the solution provided in this application embodiment, which already has good performance, breaks through the bottleneck of the key algorithm indicator IDs (ID transitions) again, and reduces it by 69%;

[0207] Specifically, such as Figure 21 The example shows a comparison of the effects in a business scenario. Before optimization, the trajectory was broken due to missed detection of pre-labeling. After optimization, the algorithm provided in this application reconnects the broken trajectory.

[0208] like Figure 22 Another complex business scenario was selected, and the full-time segment tracking trajectory of the scenario was drawn using an absolute coordinate system. Each node represents a 3D bounding box, and each line segment with a node represents a trajectory. This scenario is relatively complex, and before optimization, there were problems of broken and incorrectly connected trajectories. After optimization, the algorithm provided in this application embodiment reconnected the broken trajectories and disconnected and reconnected the incorrectly connected trajectories to the correct trajectories. This scenario demonstrates the superior performance of the method provided in this application embodiment.

[0209] The above results all indicate that the method provided in this application embodiment can effectively improve the quality of laser 3D multi-target tracking trajectory and obtain more complete and high-quality (fewer breakpoints) trajectories.

[0210] This application also provides a computer program product, which includes a program that, when run by an electronic device, causes the electronic device to perform the method described in any of the above embodiments. For example, the program product may be an application program, or an upgrade package or feature plugin for upgrading the application program, etc.

[0211] This application also provides an electronic device, the electronic device including: one or more processors; a memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any of the above embodiments.

[0212] Specifically, the electronic device can be a server or a cluster of servers that provides operator sequence optimization services. See also Figure 23 The hardware structure of electronic devices can include: processor, memory, and chipset.

[0213] The processor is the core that handles all computing tasks and requests, and includes at least the CPU.

[0214] Storage: This includes RAM and hard disks. The larger the capacity, the more requests and files can be processed simultaneously.

[0215] Chipset: Includes the Northbridge chip and Southbridge chip, which connect various I / O devices. The Northbridge chip connects to the CPU, memory, and PCI bus, while the Southbridge chip connects to I / O devices such as hard drives and network devices.

[0216] I / O Bus: Connects various I / O devices, such as network cards and USB ports. The larger the I / O bus bandwidth of the server, the more requests it can handle. I / O Devices: Include network cards, USB ports, etc. The network card is the main device for connecting the server to the external network, and its speed and performance directly affect the server's response speed and transmission speed.

[0217] Power supply: Provides stable power to the server to ensure its normal operation.

[0218] Chassis: The outer casing of server hardware, protecting it from dust, vibration, and temperature.

[0219] For example, a processor may include one or more processing units, such as a neural network processing unit (NPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution.

[0220] The memory can be used to store executable program code, including instructions. Internal memory may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. The data storage area may store data created during the use of the electronic device (such as input data, output data, etc.). Furthermore, internal memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in memory located within the processor.

[0221] Alternatively, electronic devices can also be terminal devices with a certain computing power, such as mobile phones, personal computers (PCs) (e.g., laptops), tablet PCs, wearable electronic devices, etc. Terminal devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, etc.

[0222] It is understood that the structures illustrated in the embodiments of the present invention are merely examples and do not constitute a limitation on the electronic device. The electronic device in the embodiments of this application may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0223] This application also provides a storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any of the above embodiments.

[0224] This application also provides a chip system, including a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the methods described in any of the above embodiments.

[0225] It should be noted that the solutions provided in the embodiments of this application include any one or more of the following: tracking information processing methods, program products, electronic devices, and chip systems. Alternatively, they may be other forms of software products, hardware products, or virtual service products obtained based on the technical concept of the methods described in this application specification.

[0226] In the description of the apparatus in the embodiments of this application, various modules are constructed in the apparatus to implement the function of the method flow. The division of modules is only a logical function division. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0227] Specifically, in actual implementation, the various functional modules can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; entirely in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip within the electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0228] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0229] Specifically, in one embodiment of this application, the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method steps described in the embodiment of this application.

[0230] Specifically, in one embodiment of this application, the processor of the electronic device may be a device-on-a-chip (SoC), which may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU). Specifically, in one embodiment of this application, the processor of the electronic device may be a PWM control chip.

[0231] Specifically, in one embodiment of this application, the processor may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPUs), and image signal processors (ISPs). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASICs, or one or more integrated circuits for controlling the execution of the program of the technical solution of this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0232] Specifically, in one embodiment of this application, the memory of the electronic device may be a read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer.

[0233] Specifically, in one embodiment of this application, the processor and memory can be combined into a single processing device, or more commonly, they are independent components. The processor executes program code stored in the memory to implement the method described in the embodiments of this application. In specific implementations, the memory can also be integrated into the processor, or it can be independent of the processor.

[0234] Furthermore, the devices, apparatuses, and modules described in the embodiments of this application may be implemented by computer chips or physical entities, or by products with certain functions.

[0235] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0236] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0237] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0238] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0239] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0240] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0241] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0242] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0243] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0244] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0246] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for processing tracking information based on laser point clouds, characterized in that, The method includes: Determine the confidence level corresponding to at least one first bounding box to be processed; the first bounding box is a bounding box pre-annotated to the tracked target based on the laser point cloud image; Based at least on the confidence level, reliable inputs are filtered from the at least one first bounding box to obtain a reliable input sequence; After obtaining the reliable input sequence, the method further includes: Determine N trajectories corresponding to the reliable input sequence, where N ≥ 2; each trajectory includes at least one bounding box. The m-th trajectory is sequentially input into the prediction model to obtain the first trajectory; the m-th trajectory is one of the N trajectories, 1≤m≤N; the first trajectory is the trajectory obtained by forward prediction of the m-th trajectory; The first trajectory is input into the prediction model in reverse order to obtain the second trajectory; the second trajectory is the trajectory obtained by reverse prediction of the first trajectory. Based on the first trajectory and / or the second trajectory, fill in the missing bounding boxes in the m-th trajectory and / or the second trajectory; Use a bounding box from the trusted input sequence as the anchor box; Replace the bounding boxes of different sizes in the trusted input sequence with the anchor boxes of the same size; Determine the virtual anchor point corresponding to each anchor frame; the virtual anchor point is the center point of the corresponding anchor frame.

2. The method as described in claim 1, characterized in that, Determine the confidence level corresponding to at least one first bounding box to be processed, including: Input data corresponding to at least one first bounding box to be processed is input into the prediction model, and at least one second bounding box is determined based on the output of the prediction model; Calculate the Intersection over Union (IOU) value between the first bounding box and the corresponding second bounding box, and use it as the confidence level of the first bounding box.

3. The method as described in claim 1 or 2, characterized in that, Based at least on the confidence level, filtering trusted inputs from the at least one first bounding box includes: Boundaries with a confidence level greater than or equal to a first threshold are added to the trusted input sequence.

4. The method as described in claim 3, characterized in that, Filtering trusted inputs from the at least one first bounding box based at least on the confidence level further includes: For a third bounding box whose confidence level is less than a first threshold among at least one first bounding box, perform the following operations: Calculate the iterative nearest neighbor (ICP) registration degree between the current frame and the previous or next frame corresponding to the third bounding box; The third bounding box with a confidence level less than the first threshold and greater than or equal to the second threshold, and an ICP registration level greater than or equal to the third threshold, is added to the trusted input sequence.

5. The method as described in claim 4, characterized in that, After calculating the iterative nearest neighbor (ICP) registration degree between the current frame and the previous and / or next frame corresponding to the third bounding box, the method further includes: Bounding boxes with a confidence level less than the second threshold and an ICP registration level lower than the third threshold are identified as bounding boxes to be corrected. The bounding box to be corrected is corrected; the corrected bounding box is added to the trusted input sequence.

6. The method as described in claim 1, characterized in that, After obtaining the second trajectory, the method further includes: Calculate the matching degree between the second trajectory and N-1 trajectories; the N-1 trajectories are the trajectories other than the m-th trajectory among the N trajectories; If the matching degree between the nth trajectory and the second trajectory meets a preset condition, the nth trajectory is determined to be the same trajectory as the first trajectory, and the first trajectory and the nth trajectory are connected; where n ≠ m, and 1 ≤ n ≤ N.

7. The method as described in claim 6, characterized in that, Calculate the matching degree between the second trajectory and N-1 trajectories, including: Generate a global time-series lookup table; the global time-series lookup table contains the number and timestamp corresponding to each bounding box; Calculate the IOU value between the second trajectory and the bounding boxes corresponding to the same timestamp in the N-1 trajectories, according to the timestamp from back to front. If the matching degree between the nth trajectory and the second trajectory meets a preset condition, the nth trajectory is determined to be the same trajectory as the first trajectory, including: If there are x consecutive bounding boxes between the nth trajectory and the second trajectory with an IOU value greater than the fourth threshold, then the nth trajectory and the mth trajectory are determined to be the same trajectory.

8. A computer program product, characterized in that, The program product includes a program that, when run by an electronic device, causes the electronic device to perform the method as described in any one of claims 1-7.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; memory; at least one application program; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-7.

10. A storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-7.

11. A chip system, characterized in that, include: A communication interface for inputting and / or outputting data; A processor for executing a computer-executable program, causing a device having the chip system mounted to perform the method as described in any one of claims 1-7.

Images