Target tracking method and device and computer program product

Abstract

The invention relates to a target tracking method, a target tracking device and a computer program product. The target tracking method comprises the steps of performing target detection to obtain a detection frame of each target in a point cloud frame; performing prediction based on the existing target trajectory to obtain a trajectory point prediction frame of each existing target at the current moment; associating the track point prediction frame of each existing target with the detection frame of each target; when any track point prediction frame has an associated detection frame, updating the confidence coefficient of the existing target track corresponding to the track point prediction frame according to the intersection-to-union ratio between the track point prediction frame and the detection frame; when any track point prediction frame does not have the associated detection frame, the confidence coefficient of the track corresponding to the track point prediction frame is attenuated; and determining whether an existing target trajectory is reserved according to the confidence of the updated or attenuated trajectory, and outputting a target tracking result according to the currently reserved trajectory. According to the invention, the consistency of the track confidence and the track quality can be improved.

Description

Technical Field

[0001] The present application relates to the technical field, and in particular to a target tracking method and device, and a computer program product. Background Art

[0002] Using laser point clouds for target tracking is an important research direction in the fields of autonomous driving, unknown space detection, and urban component extraction in real 3D scenes, and has been vigorously developed in recent years. As a downstream task of target detection, target tracking can provide strong support for the decision-making and planning of unmanned systems.

[0003] The multi-target tracking method based on trajectory confidence is a technology used to determine the continuity and reliability of trajectories in a multi-target tracking system. In this method, each trajectory is assigned a confidence level, which reflects the quality of the trajectory. The higher the confidence level, the higher the trajectory quality. The accurate calculation of trajectory confidence is the key technology to achieve this method. If there is a problem of inconsistency between trajectory confidence and trajectory quality, the tracking trajectory will disappear early or late. Summary of the invention

[0004] The purpose of this application is to propose a target tracking method and its device, and a computer program product to improve the consistency of trajectory confidence and trajectory quality.

[0005] To achieve the above object, according to a first aspect of the present application, a target tracking method is provided, the method comprising:

[0006] Obtaining the point cloud frame at the current moment and performing target detection to obtain the detection frame of each target in the point cloud frame;

[0007] Based on each existing target trajectory, the trajectory point prediction of the existing target at the current moment is performed to obtain the trajectory point prediction box of each existing target at the current moment;

[0008] Associating the trajectory point prediction box of each existing target with the detection box of each target;

[0009] When any trajectory point prediction box has an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is updated according to the intersection-and-union ratio between the trajectory point prediction box and the detection box;

[0010] When any trajectory point prediction box does not have an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is attenuated;

[0011] Whether to retain the existing target trajectory is determined according to the confidence of the existing target trajectory after update or attenuation, and the tracking result of the existing target is output according to the currently retained existing target trajectory.

[0012] According to the method described in the first aspect, based on the existing target trajectory, the trajectory point prediction box of the existing target at the current moment is predicted, and these prediction boxes are associated with the target box detected in the current point cloud frame. This association ensures that the continuity of the trajectory is based on the actual detected target. The prediction accuracy is evaluated by the intersection over union (IoU) between the associated trajectory point prediction box and the detection box, and the confidence of the trajectory is updated. If the IoU is high, it means that the trajectory prediction is more accurate and the trajectory confidence will be improved; conversely, if the IoU is low, the confidence will be reduced accordingly. This method ensures that the trajectory confidence can reflect the real-time quality of the trajectory. For the trajectory point prediction box that has no detection box associated with it, it is chosen to attenuate its confidence (i.e., reduce the confidence). This attenuation mechanism helps to timely eliminate those trajectories that have become invalid or have degraded in quality. Finally, it is decided whether to retain the trajectory based on the updated or attenuated confidence, which ensures that only those trajectories with higher confidence are retained, thereby improving the overall quality of the trajectory.

[0013] According to a second aspect of the present application, a target tracking device is provided, comprising a module for executing the target tracking method according to the first aspect.

[0014] According to a third aspect of the present application, a target tracking device is provided, comprising:

[0015] A communication interface, used to communicate with other electronic devices;

[0016] a memory for storing computer program instructions;

[0017] A processor is used to execute the computer program instructions to support the target tracking device according to the first aspect to implement the method as described above.

[0018] According to a fourth aspect of the present application, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions instruct a computer device to perform operations corresponding to the method according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 This is a flow chart of a target tracking method in one embodiment of the present application.

[0021] Figure 2A schematic diagram of a method for obtaining a detection frame for each target in a point cloud frame in one embodiment of the present application.

[0022] Figure 3 This is a scatter plot between the confidence of the detection box before optimization and IoU (trajectory point prediction box and detection box) in one embodiment of the present application.

[0023] Figure 4 This is a scatter plot between the confidence of the optimized detection box and IoU (trajectory point prediction box and detection box) in one embodiment of the present application.

[0024] Figure 5 This is a flowchart of a target tracking method in a specific embodiment of the present application. DETAILED DESCRIPTION

[0025] The detailed description of the drawings is intended as an illustration of the current embodiment of the present application, and is not intended to represent the only form in which the present application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included in the spirit and scope of the present application.

[0026] An embodiment of the present application provides a target tracking method, see Figure 1 , the method of this embodiment includes the following steps:

[0027] Step S10, obtaining the point cloud frame at the current moment and performing target detection to obtain the detection frame of each target in the point cloud frame.

[0028] Specifically, a point cloud is a collection of points in space, each of which has its three-dimensional coordinates and can also contain color or other information. A point cloud frame refers to a frame of 3D point cloud data of an environment captured at a specific point in time. In autonomous driving applications, LIDAR or other 3D sensors can be used to obtain point cloud data in the environment. The current moment refers to the time point that the system is processing.

[0029] Object detection refers to the process of using computer vision or machine learning algorithms to identify specific targets (such as vehicles, pedestrians, obstacles, etc.) in a point cloud frame. The algorithm analyzes the point cloud data to determine which point sets belong to independent targets and classifies each target. The bounding box is a virtual box used to mark the position and size of the detected target. In a 3D point cloud, the bounding box is three-dimensional and defined by the minimum and maximum x, y, and z coordinates. During the object detection process, the algorithm not only identifies the target, but also outputs a box to surround the target for subsequent processing, such as tracking, classification, or obstacle avoidance.

[0030] Step S20, predicting the trajectory points of the existing targets at the current moment based on the trajectories of the existing targets, and obtaining the trajectory point prediction boxes of the existing targets at the current moment.

[0031] Specifically, an existing target trajectory refers to the historical motion path of a target that the system has previously identified and tracked. Each target has a corresponding trajectory that records the position of the target at consecutive time points and other related information.

[0032] Trajectory point prediction estimates the current position of a target based on its past motion patterns. This process uses motion models to predict the next position of the target in time. Motion models include but are not limited to Kalman filtering, particle filtering, or prediction models based on deep learning. The prediction takes into account factors such as the target's speed, acceleration, direction, and possible motion constraints (for example, the vehicle cannot suddenly jump vertically).

[0033] The trajectory point prediction box is a virtual bounding box that represents the position and size of the target predicted by the algorithm at the current moment. This prediction box is calculated based on the historical trajectory and motion model of the target. In a 3D environment, the trajectory point prediction box is a cube or other 3D shape that surrounds the predicted target position.

[0034] Step S30, associating the trajectory point prediction box of each existing target with the detection box of each target.

[0035] Specifically, based on the processing results of step S10 and step S20, it is necessary to determine which trajectory point prediction boxes and which detection boxes correspond to the same target. In this embodiment, this process is called the association of trajectory point prediction boxes with detection boxes. The purpose of the association is to keep tracking the target between consecutive frames.

[0036] For example, the system can compare the similarity or correlation between each trajectory point prediction box and all detection boxes detected in the current frame. The similarity can be measured in a variety of ways, such as similarity based on features such as position, size, shape, or speed. Through comparison, the system tries to find the best matching detection box for each trajectory point prediction box. This matching can be based on some optimization algorithms, such as the Hungarian algorithm, which can find the best matching solution after considering all possible matching combinations. If a trajectory point prediction box does not find a matching detection box, it may mean that the target has temporarily left the field of view or is blocked.

[0037] Step S40: when any trajectory point prediction box has an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is updated according to the intersection-and-union ratio between the trajectory point prediction box and the detection box.

[0038] Specifically, IoU is an indicator that measures the degree of overlap between two bounding boxes (in this scenario, the trajectory point prediction box and the detection box). It is obtained by calculating the ratio of the overlapping area of ​​the two boxes to the total area of ​​the two boxes. The value range is between 0 and 1, where 1 indicates complete overlap and 0 indicates no overlap.

[0039] The track confidence is a quantitative representation of the accuracy and reliability of the track. In this embodiment, the confidence of the corresponding target track is updated according to the IoU value between the track point prediction box and the detection box. For example, if the IoU value is high, it means that the overlap area between the track point prediction box and the detection box is large and the prediction is very accurate. Therefore, the system will increase the confidence of the track. If the IoU value is low, it means that the overlap area between the track point prediction box and the detection box is small and the prediction is not accurate enough. The system will reduce the confidence of the track or keep the confidence unchanged. Through this process, step S40 ensures that the confidence of the track can reflect the accuracy of the track prediction, thereby helping the system to more effectively identify and track targets.

[0040] Step S50: when any trajectory point prediction box does not have an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is attenuated.

[0041] Specifically, confidence decay refers to reducing the confidence of the existing target trajectory corresponding to the prediction box of the trajectory point, because the failure to detect the target may indicate that the accuracy or reliability of the trajectory is decreasing. Attenuation can be a simple reduction operation, such as reducing a fixed value or percentage each iteration, or it can be a more complex function, depending on the specific design and requirements of the system. It should be noted that the purpose of confidence decay is to reflect the increasing uncertainty of the target trajectory. Over time, if the target continues to be undetected, the confidence of its trajectory will become lower and lower.

[0042] Step S60 , determining whether to retain the existing target trajectory according to the confidence of the existing target trajectory after update or attenuation, and outputting the tracking result of the existing target according to the currently retained existing target trajectory.

[0043] Specifically, in steps S40 and S50, the confidence of the target track has been updated (if there is a matching detection box) or decayed (if there is no matching detection box).

[0044] In step S60, the system will check the confidence of each target trajectory and decide whether to continue tracking the target based on the confidence level. Specifically, a first confidence threshold can be set. If the confidence of a target trajectory is lower than the first confidence threshold, it is determined that the target trajectory is no longer reliable, so it is decided not to track the target and remove it from the tracking list.

[0045] For those target trajectories whose confidence is higher than the first confidence threshold and are retained, the system will output the tracking results of these targets, which may include information such as the state of the target trajectory and the prediction box of the trajectory points, as well as the identity of the target (if the system supports the maintenance of the target identity). The tracking results can be used in the decision-making system of the autonomous vehicle, the alarm triggering of the monitoring system, or any application that requires real-time target location information.

[0046] In this embodiment, after experimental summary, the first confidence threshold can be set to 0.1.

[0047] In summary, the method of this embodiment predicts the trajectory point prediction box of the existing target at the current moment based on the existing target trajectory, and associates these prediction boxes with the target box detected in the current point cloud frame. This association ensures that the continuity of the trajectory is based on the actual detected target. The prediction accuracy is evaluated by the intersection over union (IoU) between the associated trajectory point prediction box and the detection box, and the confidence of the trajectory is updated. If the IoU is high, it means that the trajectory prediction is more accurate and the trajectory confidence will be improved; conversely, if the IoU is low, the confidence will be reduced accordingly. This method ensures that the trajectory confidence can reflect the real-time quality of the trajectory. For the trajectory point prediction box that has no detection box associated with it, it is selected to attenuate its confidence (i.e., reduce the confidence). This attenuation mechanism helps to timely eliminate those trajectories that have failed or have degraded in quality. Finally, it is decided whether to retain the trajectory based on the updated or attenuated confidence, which ensures that only those trajectories with higher confidence are retained, thereby improving the overall quality of the trajectory.

[0048] In some specific embodiments, the step S10 further includes:

[0049] The point cloud frame at the current moment is obtained and target detection is performed, and the confidence of the detection frame of each target in the point cloud frame is also obtained.

[0050] Specifically, the sensor is used to capture the environmental point cloud data at the current moment, and each target in the point cloud is identified by a target detection algorithm, a detection frame is generated for each target, and a confidence value is assigned to each detection frame, which indicates the degree of confidence of the algorithm that the detection frame contains the target. The confidence value is usually a value between 0 and 1, where 1 indicates that the algorithm is very confident that the detection frame does contain a target, and 0 indicates almost no confidence. The method also includes:

[0051] Step S70: When any detection frame does not have an associated trajectory point prediction frame, determine whether to generate a new target trajectory corresponding to the detection frame according to the confidence of the detection frame.

[0052] Specifically, during the data association process of step S30, some detection frames do not find matching trajectory point prediction frames. These detection frames may correspond to newly appeared targets, or they may not be detected because the previously tracked targets temporarily left the field of view. In view of this, this embodiment checks the confidence of each detection frame that fails to be associated with the trajectory point prediction frame in step S30. If the confidence of the detection frame is higher than the preset second confidence threshold, it is determined that the detection frame reliably represents a real target, and thus a new target trajectory is generated. This new trajectory will start from this detection frame and develop with the update of subsequent frames. If the confidence of the detection frame is lower than the second confidence threshold, it is considered that the detection frame is not reliable enough to start a new tracking. In this case, the system will ignore this detection frame or require more evidence (for example, detection in several consecutive frames) to confirm it.

[0053] In this embodiment, after experimental summary, the second confidence threshold can be set to 0.5.

[0054] By considering the confidence of the detection box, the system can reduce the error tracking caused by false detection and improve the accuracy of tracking. The system does not generate a track for each low-confidence detection box, which helps to reduce unnecessary calculation and storage requirements. In some specific embodiments, the method also includes:

[0055] Step S80, after the new target trajectory is generated, the new target trajectory is saved, and the tracking result of the new target is output according to the new target trajectory.

[0056] Specifically, according to the decision in step S70, the system has determined that a new target track has been generated for a detection box, and the new target track represents a target that the system considers to be newly appeared. The system will record the detailed information of the new target track in the memory or database so as to continue to track the target in subsequent frames. The saved information may include but is not limited to the track ID, the detection box information, the track confidence, etc. In the processing of a point cloud frame, the new target track is used as the existing target track.

[0057] Furthermore, the system will output the tracking result based on the information of the new target trajectory. The tracking result may include information such as the state and detection frame of the new target trajectory, as well as the identity of the target (if the system supports the maintenance of the target identity). The detection frame of the new target is used as the first trajectory point prediction frame of the new target trajectory. The tracking result can be used for the decision-making system of the autonomous driving vehicle, the alarm triggering of the monitoring system, or any application that requires real-time target location information. In some specific embodiments, the step S10 further includes:

[0058] In step S101, the point cloud frame is processed based on a pre-trained target detection model to obtain an initial target detection result and a bird's-eye view feature map; wherein the initial target detection result includes an initial detection frame of each target in the point cloud frame and its confidence.

[0059] Specifically, in this embodiment, a trained 3D target detection model is used to analyze the point cloud frame. This target detection model is specifically used to identify and locate targets from point cloud data to obtain initial target detection results and a bird's-eye view feature map (BEV feature map). It should be noted that the specific type of 3D target detection model selected depends on the specific application scenario, computing resources, and performance requirements, and is therefore not specifically limited in this embodiment.

[0060] The initial target detection result can be understood as the preliminary analysis result of the point cloud frame by the method of this embodiment, including the initial detection box of each target in the point cloud frame (i.e., the preliminary estimated target position and size) and its confidence (i.e., the degree of trust of the target detection model in the detection result).

[0061] The bird's-eye view feature map is a feature map generated by a target detection model, which extracts information that is helpful for target detection from point cloud data and represents this information in a form that is more suitable for subsequent processing. The bird's-eye view feature map is generated from a top view of the original point cloud data.

[0062] In step S102, the initial target detection result is used as a query vector (Query), the bird's-eye view feature map is used as a key vector (Key) and a value vector (Value), and is input into a pre-trained decoder to optimize the initial target detection result to obtain the detection box and confidence level of each target in the point cloud frame.

[0063] Specifically, the decoder can be a decoder of the Transformer architecture. In this embodiment, the initial target detection result is used as a query vector and the bird's-eye view feature map is used as a key vector and a value vector, and is input into the decoder. The decoder will adjust the initial detection result based on the query vector, the key vector and the value vector to improve the accuracy of the detection frame and the reliability of the confidence. After optimization processing by the decoder, a more accurate target detection frame and confidence are finally obtained. These optimized detection results will be used for subsequent tracking and decision-making processes. In summary, the method of this embodiment improves the accuracy and efficiency of target detection by combining the initial detection results and the bird's-eye view feature map and optimizing it using the decoder.

[0064] In some specific embodiments, see Figure 2, the decoder includes a multi-head self-attention layer, a first addition and normalization layer, a multi-head cross attention layer, a second addition and normalization layer, a feedforward network, a third addition and normalization layer and a detection head;

[0065] The multi-head self-attention layer is used to process the query vector based on the multi-head self-attention mechanism.

[0066] Specifically, in the Transformer architecture model, position encoding is a mechanism for providing position information of the input sequence. This embodiment uses the pixel position of the center point of the detection box on the BEV feature map to initialize the position encoding of the query. Specifically, in the point cloud data, each target is represented by a detection box, which contains the boundary of the target. The center point of the detection box is the geometric center of the box, which usually represents the position of the target in 3D space. The BEV feature map is formed by projecting the point cloud data onto a two-dimensional plane, which is usually a horizontal plane and is used to simulate the bird's-eye view from above. In the BEV feature map, each point corresponds to a point in the point cloud, and each pixel on the BEV feature map corresponds to a grid unit. In the Transformer model, Query, Key and Value are three key vectors, which are used in the model to implement the self-attention mechanism and the cross-attention mechanism. The position encoding of the query refers to providing each query vector with a code representing its position. By using the pixel position of the center point of the detection box on the BEV feature map as the position encoding of the query, the model can better understand the position of the query in the BEV feature map and perform attention calculation based on it.

[0067] The multi-head self-attention mechanism is a deep learning technique that allows the model to focus on different parts of the input sequence at the same time. In the multi-head self-attention layer, the input sequence (in this case, the query vector) is split into multiple subsequences, each of which is assigned an independent weight, and then the weighted sum of these subsequences is merged back into the original sequence. The multi-head self-attention layer can help the model better understand the internal relations of the input data, which is particularly useful for processing complex spatial relations in point cloud frames.

[0068] The first addition and normalization layer is used to add and normalize the query vector and the output of the multi-head self-attention layer.

[0069] Specifically, the first addition and normalization layer adds the output of the multi-head self-attention layer and the query vector to combine the information of the two; the summed result is normalized to maintain the stability of the data; the first addition and normalization layer can further enhance the model's ability to process input data.

[0070] The multi-head cross attention layer is used to process the output of the first sum normalization layer and the key vector and the value vector based on the multi-head cross attention mechanism;

[0071] Specifically, unlike the self-attention mechanism, the cross-attention mechanism focuses on the relationship between the input sequence (the output of the first addition and normalization layer) and another sequence (key vector and value vector); by focusing on the relationship between the input sequence and the key vector and value vector, the multi-head cross-attention layer can help the model understand the position and context of the input sequence in the entire environment.

[0072] The second addition and normalization layer is used to add and normalize the output of the first addition and normalization layer and the output of the multi-head cross attention layer.

[0073] Specifically, the second addition and normalization layer adds the output of the first addition and normalization layer with the output of the multi-head cross attention layer to combine the information of the two; the summed result is normalized to maintain the stability of the data; the second addition and normalization layer can further enhance the model's ability to process input data.

[0074] The feedforward network is used to perform nonlinear processing on the output of the second addition and normalization layer.

[0075] Specifically, the feedforward network can be composed of two or more fully connected layers, each of which uses ReLU (Rectified Linear Unit) as the activation function. This structure can introduce nonlinearity to the model and help the model learn more complex feature representations. In the Transformer model, the feedforward network is usually used to process the output of the self-attention mechanism and the cross-attention mechanism to further enhance the expressiveness of the model.

[0076] The third addition and normalization layer is used to add and normalize the output of the second addition and normalization layer and the output of the feedforward network.

[0077] Specifically, the third addition and normalization layer adds the output of the second addition and normalization layer with the output of the feedforward network to combine the information of the two; the summed result is normalized to maintain the stability of the data; the third addition and normalization layer can further enhance the model's ability to process input data.

[0078] The detection head is used to obtain the detection frame and confidence level of each target in the point cloud frame according to the output of the third addition and normalization layer.

[0079] Specifically, the detection head is responsible for extracting the final detection result, i.e., the detection box of the target and its confidence, from the feature vector output by the third addition and normalization layer; in this embodiment, the detection head includes a classification branch and a regression branch, the classification branch outputs the optimized detection box confidence, and the regression branch outputs the optimized detection box.

[0080] Furthermore, this embodiment uses the Sincosine encoding method shown in the following formula to generate: Query feature and position encoding This initialization method uses the detection results and features of the first stage as prior knowledge, which can make the decoder converge more easily during the training process;

[0081]

[0082]

[0083] Among them, pos is the position information, i is the i-th encoding dimension, d model is the dimension of the encoding, sin represents the sine function, and cos represents the cosine function.

[0084] Specifically, Figure 3 and Figure 4 The optimization effect is shown. Figure 3 The figure shows a scatter plot of the detection box confidence and IOU value before optimization, that is, the initial detection box output by the 3D object detection model and its IOU value with the trajectory point prediction box. Figure 4 The figure shows a scatter plot of the optimized detection box confidence and IOU value, that is, the detection box output by the decoder and its IOU value with the trajectory point prediction box.

[0085] In some specific embodiments, the predicting of trajectory points of existing targets based on the trajectories of the existing targets further includes:

[0086] A Kalman filter model is used to predict the trajectory points of an existing target. In the Kalman filter model, the trajectory point state quantity of the existing target trajectory includes the three-dimensional coordinates, length, width and height of the center point position of the trajectory point prediction box, and the heading angle and moving speed of the existing target;

[0087] Among them, the heading angle and moving speed of the existing target are obtained according to the position of the existing target in the two previous and subsequent point cloud frames. Specifically, the heading angle can be calculated by comparing the angle change between the positions of the target in the two point cloud frames, and the moving speed can be calculated by comparing the distance change between the positions of the target in the two point cloud frames.

[0088] Specifically, the Kalman filter model mainly includes a state equation and an observation equation. The specific steps of initializing the target trajectory using the Kalman filter algorithm in this embodiment are as follows:

[0089] (step1) Initialize the state equation:

[0090] First, define the state quantity of each trajectory as Indicates its state at time t, and n is the number of state quantities. When tracking a point cloud 3D target, its state quantity is:

[0091] x t =[p x ,p y ,p z ,l,w,,θ,v x ,v y ,v z ];

[0092] Among them, p x ,p y ,p z Represents the three-dimensional coordinates of the center point of the target box represented by the trajectory, l, w, respectively represent the length, width, and height of the trajectory point prediction box, θ is the heading angle of the target, and v x ,v y ,v z They represent the moving speed components of the target in three-dimensional directions respectively;

[0093] Based on the above definition, a state estimation model can be established, which is as follows:

[0094] x t =F t x t-1 +p t

[0095]

[0096] in, is the state transfer matrix, Δt is the time difference between time t and time t-1, is a Gaussian noise matrix used to characterize the uncertainty of the state model, x t-1 Indicates the state of the target trajectory in the previous frame.

[0097] (step2) Initialize the observation equation:

[0098] The observation equation is used to obtain the observation value corresponding to the trajectory from the state quantity at the current moment (that is, the position and size of the target box predicted by the trajectory at the current moment). The formula is as follows:

[0099] z t =H tx t +v t

[0100]

[0101] In the formula, is the observation matrix, v t For R t Gaussian noise matrix.

[0102] Furthermore, after initializing the state equation and observation equation of the target trajectory, the trajectory prediction can be performed to obtain the trajectory point prediction box corresponding to the current moment. According to the state equation and the trajectory state quantity x at the previous moment, t-1 The trajectory state x at the current moment can be calculated t , according to the calculation, we get x t You can update z t , and then perform Kalman gain update, according to the calculated z t The final estimated trajectory point state can be obtained by adding the Kalman gain. Based on the final estimated trajectory point state, the corresponding trajectory point prediction box can be obtained, that is, the prediction box containing p x ,p y ,p z ,l,w,and other parameters.

[0103] In some specific embodiments, associating the trajectory point prediction box of each existing target with the detection box of each target further includes:

[0104] When the intersection-and-union ratio between any trajectory point prediction frame and at least one detection frame is greater than a preset intersection-and-union ratio threshold, it is determined that any trajectory point prediction frame has an associated detection frame;

[0105] All trajectory point prediction frames with associated detection frames are obtained, and a preset matching algorithm is used to match an associated detection frame for all the trajectory point prediction frames; wherein the optimization purpose of the preset matching algorithm is to seek the minimum overall matching cost.

[0106] Specifically, the present embodiment pre-sets an intersection-over-union threshold IoU_t, and the intersection-over-union ratio between the trajectory point prediction box and the detection box is IoU_3d. When IoU_3d>IoU_t, it indicates that the trajectory point prediction box obtained by tracking the trajectory is associated with the detection box output by the target detection model, otherwise there is no association between the two. When the distribution positions of the trajectory point prediction box and the detection box are relatively concentrated, there may be a trajectory point prediction box and multiple detection boxes whose IoU_3d values ​​are all greater than IoU_t. Therefore, in order to select the detection box with the highest correlation for each trajectory point prediction box, the present embodiment pre-sets a preset matching algorithm as the matching algorithm between the trajectory point prediction box and the detection box, such as the Hungarian Algorithm, or a distance-based matching method.

[0107] The optimization purpose of the matching algorithm is to minimize the overall matching cost, which means that the algorithm will find a matching solution so that the sum of the matching costs of all matching pairs is minimized. The matching cost can be based on multiple factors, such as the intersection over union (IoU_3d) between the trajectory point prediction box and the detection box, that is, the sum of the IoU_3d of the overall matching result is maximized.

[0108] The matching process is as follows:

[0109] When it is found that the intersection-and-union ratio of at least one trajectory point prediction box and the detection box is greater than the preset intersection-and-union ratio threshold, it is determined that the trajectory point prediction box has an associated detection box. Then, the system obtains all trajectory point prediction boxes that have associated detection boxes. Finally, a preset matching algorithm is used to match an associated detection box for all trajectory point prediction boxes to ensure that each prediction box has a correct match.

[0110] In this way, it is ensured that each track point prediction box is correctly matched with a detection box, thereby improving the accuracy and reliability of target tracking. This matching method is particularly important when dealing with complex and dynamically changing scenes because it helps the system maintain continuous tracking of the target even if the target moves or is partially occluded between frames.

[0111] Figure 5A specific embodiment of the present application is shown. In this specific embodiment, the detection frame and its confidence of each target in the point cloud frame are first obtained, as well as each trajectory point prediction frame obtained by Kalman filtering prediction based on the existing target trajectory; the IoU value between each target detection frame and each trajectory point prediction frame is calculated, and all trajectory point prediction frames and detection frames with IoU values ​​greater than the threshold IoU_t are screened out, and matched based on the Hungarian algorithm to match an associated detection frame for each trajectory point prediction frame; then, it is determined whether the IoU value between each trajectory point prediction frame and its associated detection frame is greater than 0.1. If not, the trajectory confidence is attenuated. If so, the trajectory confidence is updated; if the attenuated or updated trajectory confidence is greater than 0.1, the trajectory is retained, and if it is less than or equal to 0.1, the trajectory is discarded; for the detection frame without an associated trajectory point prediction frame, it is determined whether its confidence is greater than 0.5. If so, a new target trajectory is generated. If not, it is determined to be an invalid detection frame; finally, the tracking result is output based on the retained and newly generated trajectories.

[0112] Another embodiment of the present application also provides a target tracking device, including a module for executing the target tracking method described in the above embodiment. This device can be a hardware device, a software device, or a combination of hardware and software.

[0113] Specifically, this device includes a series of modules, which can execute each step in the target tracking method described above, for example: a module is used to obtain a point cloud frame at the current moment and perform target detection to obtain a detection frame of each target in the point cloud frame; a module is used to predict the trajectory points of the existing target at the current moment based on each existing target trajectory to obtain the trajectory point prediction frame of each existing target at the current moment; a module is used to associate the trajectory point prediction frame of each existing target with the detection frame of each target; a module is used to update the confidence of the existing target trajectory corresponding to the trajectory point prediction frame according to the intersection and union ratio between the trajectory point prediction frame and the detection frame when any trajectory point prediction frame has an associated detection frame; a module is used to attenuate the confidence of the existing target trajectory corresponding to the trajectory point prediction frame when any trajectory point prediction frame does not have an associated detection frame; a module is used to determine whether to retain the existing target trajectory according to the confidence of the updated or attenuated existing target trajectory, and output the tracking result of the existing target according to the currently retained existing target trajectory.

[0114] Furthermore, the trajectory confidence is updated according to the following formula:

[0115]

[0116]

[0117] Among them, t represents the IoU value (IoU) between the prediction box and the detection box based on the trajectory point at the current moment. t ) can be calculated as s t , use s t The confidence c of the trajectory at the previous moment t-1 Jointly update to get the trajectory confidence c at the current moment t , α is a preset parameter value, which can be set according to the technical requirements of the application scenario.

[0118] Furthermore, the trajectory confidence is attenuated according to the following formula:

[0119] c t =c t-1 -0.1.

[0120] It should be noted that the device of this embodiment corresponds to the method of the above embodiment. Therefore, the specific principle content of the module of the device of this embodiment can be obtained by referring to the content of the method of the above embodiment, so it will not be repeated here.

[0121] Another embodiment of the present application also provides another target tracking device, including:

[0122] A communication interface, used to communicate with other electronic devices;

[0123] a memory for storing computer program instructions;

[0124] The processor is used to execute the computer program instructions to support the target tracking device to implement the method described in the above embodiment.

[0125] In this embodiment, the memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating device, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (FlashCard), etc., or the memory can also be other volatile solid-state storage devices.

[0126] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the target tracking device, and uses various interfaces and lines to connect the various parts of the target tracking device.

[0127] An embodiment of the present application also provides a computer program product, including computer program instructions, wherein the computer program instructions instruct a computer device to perform operations corresponding to the method described in the above embodiment.

[0128] Specifically, the computer program product includes a series of computer program instructions that can instruct a computer device to perform the target tracking method described in this application. These instructions are codes written in a computer program that define how to perform specific operations. In this embodiment, these instructions are used to perform the target tracking method of the above embodiment.

[0129] These program instructions are designed to be loaded onto a computer device and instruct the device to perform specific operations, which are the various steps in the target tracking method described in the above embodiment.

[0130] In this way, the computer program product provides a complete software solution, which can be run on various computer devices to implement the target tracking method of the above embodiment.

[0131] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A target tracking method, characterized in that: The method comprises: Obtaining the point cloud frame at the current moment and performing target detection to obtain the detection frame of each target in the point cloud frame; Based on each existing target trajectory, the trajectory point prediction of the existing target at the current moment is performed to obtain the trajectory point prediction box of each existing target at the current moment; Associating the trajectory point prediction box of each existing target with the detection box of each target; When any trajectory point prediction box has an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is updated according to the intersection-and-union ratio between the trajectory point prediction box and the detection box; When any trajectory point prediction box does not have an associated detection box, the confidence of the existing target trajectory corresponding to the trajectory point prediction box is attenuated; Whether to retain the existing target trajectory is determined according to the confidence of the existing target trajectory after update or attenuation, and the tracking result of the existing target is output according to the currently retained existing target trajectory.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining the point cloud frame at the current moment and performing target detection, and also obtaining the confidence of the detection frame of each target in the point cloud frame; When any detection box does not have an associated trajectory point prediction box, it is determined whether to generate a new target trajectory corresponding to the detection box based on the confidence of the detection box.

3. The method according to claim 2, characterized in that The method further comprises: After the new target trajectory is generated, the new target trajectory is saved, and the tracking result of the new target is output according to the new target trajectory.

4. The method according to claim 2, characterized in that: Acquiring the point cloud frame at the current moment and performing target detection to obtain the detection frame of each target in the point cloud frame and its confidence, further includes: The point cloud frame is processed based on a pre-trained target detection model to obtain an initial target detection result and a bird's-eye view feature map; wherein the initial target detection result includes an initial detection frame of each target in the point cloud frame and its confidence; The initial target detection result is used as a query vector, and the bird's-eye view feature map is used as a key vector and a value vector, and is input into a pre-trained decoder to optimize the initial target detection result to obtain the detection box and confidence level of each target in the point cloud frame.

5. The method according to claim 4, characterized in that The decoder comprises: A multi-head self-attention layer, used for processing the query vector based on a multi-head self-attention mechanism; A first summing and normalizing layer, used to sum and normalize the query vector and the output of the multi-head self-attention layer; A multi-head cross attention layer, used for processing the output of the first summing and normalization layer and the key vector and the value vector based on a multi-head cross attention mechanism; A second addition and normalization layer, used for adding and normalizing the output of the first addition and normalization layer and the output of the multi-head cross attention layer; A feed-forward network, configured to perform nonlinear processing on the output of the second addition and normalization layer; a third addition and normalization layer, used for adding and normalizing the output of the second addition and normalization layer and the output of the feedforward network; A detection head is used to obtain a detection frame and a confidence level of each target in the point cloud frame according to an output of the third addition and normalization layer.

6. The method according to claim 1, characterized in that The step of predicting the trajectory points of existing targets based on the trajectories of the existing targets further includes: A Kalman filter model is used to predict the trajectory points of an existing target. In the Kalman filter model, the trajectory point state quantity of the existing target trajectory includes the three-dimensional coordinates, length, width and height of the center point position of the trajectory point prediction box, and the heading angle and moving speed of the existing target; wherein the heading angle and moving speed of the existing target are obtained according to the position of the existing target in the two previous and next point cloud frames.

7. The method according to claim 2, characterized in that The associating the trajectory point prediction frame of each existing target with the detection frame of each target further includes: When the intersection-and-union ratio between any trajectory point prediction frame and at least one detection frame is greater than a preset intersection-and-union ratio threshold, it is determined that any trajectory point prediction frame has an associated detection frame; All trajectory point prediction frames with associated detection frames are obtained, and a preset matching algorithm is used to match an associated detection frame for all the trajectory point prediction frames; wherein the optimization purpose of the preset matching algorithm is to seek the minimum overall matching cost.

8. A target tracking device, characterized in that: Comprising modules for executing the method according to any one of claims 1 to 7.

9. A target tracking device, characterized in that: include: A communication interface, used to communicate with other electronic devices; a memory for storing computer program instructions; A processor, configured to execute the computer program instructions to support the apparatus to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer program instructions, wherein the computer program instructions instruct a computer device to execute operations corresponding to the method according to any one of claims 1 to 7.

Images