Target tracking method, apparatus and device, and computer program product

Abstract

The invention discloses a target tracking method, apparatus and device, and a computer program product. The target tracking method comprises the steps of obtaining a multi-view image of a current frame collected by a multi-view camera; according to the multi-view image of the current frame, performing target detection by using a BEV detection model to obtain a 3D target detection result of the current frame and a corresponding BEV feature; according to the 3D target detection result of the current frame, performing time sequence tracking by using a target tracking algorithm to obtain an initial target tracking result; and according to the initial target tracking result and the BEV features, correcting the initial target tracking result by using a geometric regression model to obtain a corrected target tracking result. According to the method, pure visual perception is taken as a core, a BEVFormer model is utilized to map a multi-view image to an aerial view space, and global scene information and object geometric features are effectively fused; and the GRM model is used for performing time sequence consistency optimization on the size of the target, so that the target tracking precision and the environmental adaptability in a complex dynamic scene are improved.

Description

Technical Field

[0001] This application relates to the field of target detection and tracking technology, and in particular to a target tracking method, apparatus and equipment, and computer program product. Background Technology

[0002] With the rapid development of autonomous driving technology, target detection and tracking, as one of the key technologies in autonomous driving perception systems, has received widespread attention. In the fields of autonomous driving and intelligent transportation, target tracking technology has been widely applied based on multi-sensor fusion. Multi-sensor fusion typically combines data from sensors such as LiDAR, millimeter-wave radar, and cameras, using information provided by different sensors to track targets. However, existing technologies for target detection and tracking based on LiDAR point cloud features often have some limitations. For example, in some complex scenarios, LiDAR point cloud data may be affected by environmental factors (such as rain, snow, and changes in lighting) or limitations of the sensor itself (such as blind spots and reflection characteristics), resulting in sparse or even missing point cloud data, thus affecting the accuracy of target tracking.

[0003] In existing laser point cloud-based tracking methods, when using deep learning-based optimization algorithms to refine the target bounding box, point cloud features are typically relied upon to adjust the target's position and shape. However, in some tracklets, laser point cloud reflections may not be detected, leading to significant errors in the prediction of the target's dimensions, position, and shape, thus affecting the final tracking consistency.

[0004] Currently, although some methods attempt to supplement sparse portions of laser point clouds with visual data, most still rely on the laser point cloud to correct the bounding boxes, ignoring the potential of the pure visual data itself. Furthermore, while laser point clouds provide rich spatial information, they are costly, demanding on sensors, and computationally burdensome during data processing. In contrast, visual sensors, with their lower cost and lower sensor requirements, can provide effective information for autonomous driving systems, especially in situations where point clouds are sparse or missing, where the application of pure visual features may demonstrate unique advantages.

[0005] Therefore, how to improve tracking accuracy while maintaining the system's low cost and high efficiency when point cloud information is missing or incomplete has become a major challenge in the field of autonomous driving. Summary of the Invention

[0006] This application provides a target tracking method, apparatus, device, and computer program product to improve target tracking accuracy without relying on laser point cloud data.

[0007] The embodiments of this application adopt the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a target tracking method, the target tracking method comprising:

[0009] Acquire multi-view images of the current frame captured by a multi-view camera;

[0010] Based on the multi-view image of the current frame, target detection is performed using the BEV detection model to obtain the 3D target detection result and the corresponding BEV features of the current frame.

[0011] Based on the 3D target detection results of the current frame, a target tracking algorithm is used for temporal tracking to obtain the initial target tracking results;

[0012] Based on the initial target tracking results and the BEV features, the initial target tracking results are corrected using a geometric regression model to obtain the corrected target tracking results.

[0013] Optionally, the step of performing temporal tracking using a target tracking algorithm based on the 3D target detection result of the current frame to obtain the initial target tracking result includes:

[0014] Using the target tracking algorithm, the 3D target detection result of the current frame is matched with the existing target trajectory fragments to obtain the matching result;

[0015] Based on the matching results, the trajectory segments of existing targets are updated or new target trajectory segments are created using the 3D target detection results of the current frame.

[0016] Optionally, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The step of correcting the initial target tracking result using a geometric regression model based on the initial target tracking result and the BEV features to obtain the corrected target tracking result includes:

[0017] Determine whether the preset target tracking correction condition is triggered in the current frame based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame;

[0018] When the preset target tracking correction condition is triggered, the initial target tracking result is corrected using the geometric regression model based on the initial target tracking result and the BEV feature, to obtain the corrected target tracking result.

[0019] Optionally, determining whether the preset target tracking correction condition is triggered in the current frame based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame includes:

[0020] Based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame, determine whether the length of the target trajectory segment reaches a preset length threshold or a preset sliding window size;

[0021] If so, then the current frame is determined to trigger the preset target tracking correction condition;

[0022] Otherwise, it is determined that the current frame has not triggered the preset target tracking correction condition.

[0023] Optionally, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The step of correcting the initial target tracking result using a geometric regression model based on the initial target tracking result and the BEV features to obtain the corrected target tracking result includes:

[0024] The BEV features of each frame corresponding to the trajectory segment of the target are cropped to obtain the local BEV features of each frame.

[0025] Adaptive pooling is performed on the local BEV features of each frame to obtain the adaptively pooled local BEV features of each frame.

[0026] Based on the trajectory segment of the target and the local BEV features of each frame after adaptive pooling, the trajectory segment of the target is corrected using the geometric regression model to obtain the corrected target tracking result.

[0027] Optionally, the step of correcting the trajectory segment of the target using the geometric regression model based on the trajectory segment of the target and the local BEV features of each frame after adaptive pooling, to obtain the corrected target tracking result, includes:

[0028] The trajectory fragment of the target and the local BEV features of each frame after adaptive pooling are input into the geometric regression model to obtain the corrected target size predicted by the geometric regression model for each frame.

[0029] The target size in each frame of the trajectory segment of the target is updated using the corrected target size for each frame.

[0030] Optionally, the geometric regression model is trained in the following manner:

[0031] The training sample data for constructing the geometric regression model includes trajectory segments of the target, corresponding BEV feature sequences, and ground truth size sequences of the target.

[0032] The BEV feature sequence is cropped to obtain a local BEV feature sequence;

[0033] The local BEV feature sequence is adaptively pooled to obtain the adaptively pooled local BEV feature sequence.

[0034] The trajectory segment of the target, the adaptively pooled local BEV feature sequence, and the true size sequence of the target are input into the geometric regression model to obtain the size regression offset sequence predicted by the geometric regression model.

[0035] The total loss value of the geometric regression model is calculated based on the size regression offset sequence, and the parameters of the geometric regression model are updated using the total loss value to obtain the trained geometric regression model.

[0036] Secondly, embodiments of this application also provide a target tracking device, the target tracking device comprising:

[0037] The acquisition unit is used to acquire the multi-view image of the current frame captured by the multi-view camera;

[0038] The target detection unit is used to perform target detection using the BEV detection model based on the multi-view image of the current frame, and obtain the 3D target detection result and the corresponding BEV features of the current frame.

[0039] The target tracking unit is used to perform temporal tracking using a target tracking algorithm based on the 3D target detection results of the current frame to obtain an initial target tracking result;

[0040] The correction unit is used to correct the initial target tracking result based on the initial target tracking result and the BEV feature using a geometric regression model to obtain the corrected target tracking result.

[0041] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0042] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned target tracking methods.

[0043] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement any of the aforementioned target tracking methods.

[0044] The above-mentioned technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: The target tracking method of the embodiments of this application first acquires the multi-view image of the current frame captured by the multi-view camera; then, based on the multi-view image of the current frame, target detection is performed using a BEV detection model to obtain the 3D target detection result and the corresponding BEV features of the current frame; then, based on the 3D target detection result of the current frame, temporal tracking is performed using a target tracking algorithm to obtain the initial target tracking result; finally, based on the initial target tracking result and the BEV features, the initial target tracking result is corrected using a geometric regression model to obtain the corrected target tracking result. The target tracking method of the embodiments of this application takes pure visual perception as its core, uses the BEVFormer model to map multi-view images to a bird's-eye view space, effectively integrates global scene information and object geometric features, and avoids the high cost and complex data processing requirements of LiDAR. Based on the BEV characteristics of the target, the geometric regression model is used to optimize the temporal consistency of the target tracking size information, which reduces the problems of target size fluctuation and identity switching. Especially in the case of sparse point cloud or environmental interference, it can still maintain stable tracking performance, improve the accuracy and environmental adaptability of target tracking in autonomous driving or intelligent transportation scenarios, and provide reliable technical support for perception systems in complex dynamic scenarios. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a target tracking method according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of the structure of a target tracking device according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0051] Existing multi-object tracking methods have the following main shortcomings in constructing temporal trajectory segments (tracklets) based on 3D detection results:

[0052] (1) Problem of inconsistent target size

[0053] In traditional tracklets based on direct connection of detection results, the length, width, and height of the target can fluctuate in different frames due to insufficient detector accuracy or environmental interference. This results in a lack of consistency in the geometric properties of the same target over time, thus affecting the overall tracking accuracy.

[0054] (2) The problem of lack of high-dimensional feature constraints

[0055] Existing methods often rely on point cloud features or low-dimensional geometric state information for correction, which makes it difficult to guarantee the correction effect when the input data is missing or noisy. BEV features, on the other hand, can integrate multi-view information in a unified bird's-eye view space, providing rich geometric and semantic representations, but existing technologies have not fully utilized this advantage.

[0056] (3) Problems of insufficient time series modeling

[0057] Conventional tracking algorithms such as Kalman filtering or Hungarian matching only correlate position and velocity between frames, lacking modeling of global temporal features within the tracklet, which leads to a further decrease in the geometric consistency of the target box during long-term tracking.

[0058] (4) Problems with insufficient accuracy and stability

[0059] Due to the above factors, the final tracking results suffer from jitter, abrupt changes in geometric properties, or decreased accuracy in long-term scenarios, making it difficult to meet the needs of application scenarios with high requirements for target detection and tracking accuracy.

[0060] Based on this, embodiments of this application provide a target tracking method, such as... Figure 1 The diagram shows a flowchart of a target tracking method according to an embodiment of this application. The target tracking method includes the following steps S110 to S140:

[0061] Step S110: Obtain the multi-view image of the current frame captured by the multi-view camera.

[0062] Multiple cameras with different perspectives (e.g., eight perspectives) are deployed at intersections or in complex traffic scenarios to cover the field of view in different directions. Each camera acquires 2D image data of the current frame in real time, forming a multi-view image set. , where t represents the current time, v represents the v-th viewpoint, and V represents the number of viewpoints.

[0063] Step S120: Based on the multi-view image of the current frame, perform target detection using the BEV detection model to obtain the 3D target detection result and corresponding BEV features of the current frame.

[0064] After acquiring multi-view images of the current frame, a BEV detection model such as BEVFormer is used to perform object detection on the multi-view images. BEVFormer is a Transformer-based pure vision BEV (Bird's Eye View) perception model, mainly used for object detection and tracking tasks in autonomous driving scenarios. It captures spatial and temporal features through a self-attention mechanism, mapping multi-view 2D image features onto a 3D BEV grid, thereby achieving high-precision 3D object detection.

[0065] The BEVFormer model uses a convolutional neural network (CNN) to extract local features from images at different viewpoints, and then uses a self-attention mechanism to map the multi-view 2D features to a unified bird's-eye view (BEV) space, generating BEV feature maps. The BEV feature map preserves the global spatial structure of the scene and object geometric information, providing high-dimensional semantic support for subsequent tracking. Finally, the target is detected on the BEV feature map, and the target's 3D bounding box b is output. t,i = [x, y, z, l, w, h, ψ], where (x, y, z) is the position in the world coordinate system, (l, w, h) are the length, width and height of the target, and ψ is the heading angle.

[0066] Step S130: Based on the 3D target detection results of the current frame, perform temporal tracking using a target tracking algorithm to obtain the initial target tracking results.

[0067] After obtaining the 3D target detection results of the current frame, inter-frame temporal tracking is performed using a predefined target tracking algorithm. The predefined target tracking algorithm can be, for example, an extended Kalman filter algorithm or a linear Kalman filter algorithm. Target tracking is achieved by matching the target detection results between adjacent frames.

[0068] Considering that the Kalman filter algorithm only relies on geometric correlation and does not utilize the high-dimensional features of the target, it may lead to size fluctuations or ID switching, which in turn may result in certain errors in the tracking results. Therefore, the target tracking results obtained based on the predefined target tracking algorithm are only preliminary target tracking results and need to be further optimized by combining BEV features and geometric regression models.

[0069] Step S140: Based on the initial target tracking result and the BEV feature, the initial target tracking result is corrected using a geometric regression model to obtain the corrected target tracking result.

[0070] The initial target tracking results can include the target's trajectory segments (tracklet). For each target's tracklet and corresponding BEV features, the Geometric Refine Model (GRM) is used to perform an overall analysis of the target detection results and corresponding BEV features in all frames of the target's tracklet, and output the optimized target detection results for each frame. Here, the high-dimensional feature information of the BEV features is mainly used to correct and optimize the target size information.

[0071] The target tracking method in this application is based on pure visual perception. It utilizes the BEVFormer model to map multi-view images onto a bird's-eye view space, effectively fusing global scene information with object geometric features, thus avoiding the high cost and complex data processing requirements of LiDAR. Based on the target's BEV features, a geometric regression model is used to optimize the temporal consistency of the target's tracking size information, reducing target size fluctuations and identity switching issues. It maintains stable tracking performance, especially in scenarios with sparse point clouds or environmental interference, improving the accuracy and environmental adaptability of target tracking in autonomous driving or intelligent transportation scenarios, and providing reliable technical support for perception systems in complex dynamic scenarios.

[0072] In some embodiments of this application, the step of performing temporal tracking using a target tracking algorithm based on the 3D target detection result of the current frame to obtain an initial target tracking result includes: using the target tracking algorithm to match the 3D target detection result of the current frame with existing target trajectory segments to obtain a matching result; and based on the matching result, using the 3D target detection result of the current frame to update the existing target trajectory segments or create new target trajectory segments.

[0073] The target tracking algorithm in this application embodiment can be based on a Kalman filtering algorithm, such as an extended Kalman filter or a linear Kalman filter algorithm, to achieve target tracking. The specific implementation process is as follows:

[0074] (1) Definition of state vector and model construction

[0075] Define the state vector S of the extended or linear Kalman filter. t =[x,y,z,v x ,v y ,v z ,l,w,h,ψ] T Includes the target's three-dimensional position (x, y, z) and velocity (v). x ,v y ,v z ), dimensions (l, w, h) and orientation angle ψ.

[0076] A discrete-time linear motion model is adopted. The state transition matrix F is constructed based on the uniformity assumption. The process noise covariance Q is used to model the motion uncertainty, and the covariance matrix P initializes the uncertainty of the target state.

[0077] (2) Prediction (time update)

[0078] State prediction: through formula Predict the target state at the current moment, where u t This is an optional control variable.

[0079] Covariance prediction: Synchronously update the covariance matrix This reflects a decrease in the confidence level of the predicted state.

[0080] (3) Measurement update and data association

[0081] Observation model: Define the observation matrix H, which maps the state vector to the measurement space (e.g., observing only the position and size components). The observation noise covariance R describes the error characteristics of the detection results.

[0082] Kalman gain calculation: via Balance the weights of prediction and measurement.

[0083] State correction: Utilize the detection results of the current frame Update state estimation and correct covariance .

[0084] Data association: The Hungarian algorithm or Intersection over Union (IoU) matching is used to associate the current frame detection box with existing tracks. If a match is successful, the track is updated; if the detection box does not match any track, a new track is initialized.

[0085] (4) Track Management

[0086] Trajectory initialization: Unmatched detection boxes generate temporary trajectories, which need to be continuously matched in subsequent frames to confirm their validity.

[0087] Trajectory Termination: If no detection result is found for a trajectory in multiple consecutive frames, it is determined that the target has disappeared or the tracking has failed, and the trajectory is deleted.

[0088] Output initial tracking results: The set of detection boxes for frame t corresponding to BEVFormer is as follows Each detection box is represented as (World coordinate system), then each trajectory The set of detection boxes containing the target over time is used as the initial tracking result, where, t is the number of frames in this tracklet.j For the tth j frame.

[0089] This application's embodiments achieve efficient temporal tracking through Kalman filtering. Utilizing a closed-loop feedback mechanism of state prediction and measurement updates, it significantly improves the continuity and accuracy of target tracking. While maintaining low computational complexity, it provides preliminary tracking results for subsequent geometric regression optimization.

[0090] In some embodiments of this application, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The step of correcting the initial target tracking result using a geometric regression model based on the initial target tracking result and the BEV feature to obtain the corrected target tracking result includes: determining whether the current frame triggers a preset target tracking correction condition based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame; if the preset target tracking correction condition is triggered, correcting the initial target tracking result using the geometric regression model based on the initial target tracking result and the BEV feature to obtain the corrected target tracking result.

[0091] To ensure the correction effect of the GRM model, this embodiment of the application can first determine the target tracking correction conditions before using the GRM model to correct the initial target tracking results. The target tracking correction conditions can be predefined as whether the length or number of frames of the tracklet reaches a threshold (such as 5 consecutive frames) or meets the sliding window size requirements. The threshold and sliding window size can be dynamically adjusted according to the scene.

[0092] The length of each target's tracklet is monitored in real time. If the length of the tracklet tracked in the current frame meets the requirements of the target tracking correction condition, it can be marked as a tracklet that needs correction; otherwise, no further correction processing is performed. Conditional filtering avoids ineffective optimization of short-trajectory targets, balancing computational efficiency and tracking accuracy.

[0093] This application's embodiments significantly improve the robustness and efficiency of target tracking through the synergistic optimization of a conditional triggering mechanism and a geometric regression model. The conditional triggering mechanism, by filtering effective trajectory segments, avoids redundant calculations on short-term trajectories, optimizes resource allocation, and improves the accuracy and reliability of target tracking result correction.

[0094] In some embodiments of this application, determining whether the current frame triggers a preset target tracking correction condition based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame includes: determining whether the length of the trajectory segment of the target reaches a preset length threshold or a preset sliding window size based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame; if so, determining that the current frame triggers the preset target tracking correction condition; otherwise, determining that the current frame does not trigger the preset target tracking correction condition.

[0095] The predefined target tracking correction conditions in this application can be divided into two types: preset length threshold and preset sliding window size. The preset length threshold is mainly set for targets with short tracking lengths or few tracking frames. For example, some targets disappear after only 10 consecutive tracking frames. In this case, a preset length threshold can be used to set the tracking correction conditions. For example, if the preset length threshold is set to 5 frames, when the accumulated tracking data is less than 5 frames, it is considered that the current tracking correction conditions are not met and no correction processing is required. When 5 or more consecutive tracking frames are reached, it means that enough data frames have been accumulated for subsequent correction model analysis. That is, it is considered that the current tracking correction conditions are met, and subsequent correction processing can be performed on the data of 5 or more consecutively tracked frames.

[0096] The preset sliding window size is mainly set for targets with long tracking lengths or a large number of tracking frames. For example, if some targets are continuously tracked in the current scene for a long time (such as 30 frames), the preset sliding window size can be used to set the tracking correction conditions. For example, if the preset sliding window size is set to 5 frames, when the accumulated tracking data is less than 5 frames, it is considered that the current tracking correction conditions are not met and no correction processing is required; when 5 or more frames are continuously tracked, it is considered that the current tracking correction conditions are met, and based on the principle of sliding window, the latest 5 frames of data can be used for subsequent correction processing in real time.

[0097] Those skilled in the art can flexibly set the aforementioned preset length threshold and preset sliding window size according to the actual scenario, and no specific limitations are made here.

[0098] This application's embodiments achieve a precise balance between computational efficiency and tracking accuracy through a dynamic condition triggering mechanism. The flexible configuration of the preset length threshold and sliding window size allows the system to adapt to different scenario requirements. For example, in scenarios where the target changes rapidly, the preset length threshold enables a quick response to target changes; in scenarios where changes are slower, the preset sliding window size processes the most recent temporal information, thereby improving the efficiency of correction processing.

[0099] In some embodiments of this application, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The step of correcting the initial target tracking result using a geometric regression model based on the initial target tracking result and the BEV features to obtain the corrected target tracking result includes: cropping the BEV features of each frame corresponding to the trajectory segment of the target to obtain local BEV features of each frame; performing adaptive pooling on the local BEV features of each frame to obtain adaptively pooled local BEV features of each frame; and correcting the trajectory segment of the target using the geometric regression model based on the trajectory segment of the target and the adaptively pooled local BEV features of each frame to obtain the corrected target tracking result.

[0100] Assuming the world coordinate range covered by the BEV feature is For example, [-100, 100] x [-100, 100] m, with a BEV resolution of H x W. Given the center (x, y) and size (l, w) of the detection box, its pixel boundaries can be calculated as follows:

[0101]

[0102]

[0103]

[0104]

[0105] To enhance robustness, pixel boundaries are typically enlarged. .

[0106] Based on the magnified pixel boundaries in the BEV feature map F t Top-cropping local features (BEV-crops) yields local BEV features for each frame. Its size is dynamically determined by the detection frame.

[0107] Due to different frames, different objects, and different categories of cropping area sizes The inconsistencies need to be resolved to a fixed resolution acceptable to GRM. In this application embodiment, adaptive pooling can be used to achieve the following:

[0108] Using `torch.nn.functional.adaptive_avg_pool2d`, arbitrary-sized clipping regions are directly pooled. The pooling process preserves the semantic information of the features, avoids semantic dilution caused by explicit padding, and ensures that the features input to the GRM contain both local details of the target and global context.

[0109] The adaptive pooling features of all frames within the target's trajectory segment are arranged in chronological order to form a temporal feature sequence. This temporal feature sequence, along with the detection results (position and size predictions) of the initial trajectory, are input into the GRM model. The GRM analyzes the target's motion pattern through temporal modeling, regresses and optimizes the size (length, width, and height) and position of each frame within the trajectory, and eliminates geometric fluctuations caused by detection noise or occlusion.

[0110] In some embodiments of this application, index management and temporal consistency management can also be performed on the target detection results and the target trajectory segments. Specifically, an ID mapping table between target detection boxes and trajectory segments is established to ensure that the complete trajectory BEV-crops sequence is extracted according to the target ID during model training / inference. Features are processed strictly in chronological order to ensure that the temporal features of the GRM input are consistent with the frame order of the trajectory segments, avoiding correction errors caused by time misalignment.

[0111] In some embodiments of this application, the step of correcting the trajectory segment of the target based on the trajectory segment of the target and the local BEV features of each frame after adaptive pooling, and obtaining the corrected target tracking result, includes: inputting the trajectory segment of the target and the local BEV features of each frame after adaptive pooling into the geometric regression model to obtain the corrected target size predicted by the geometric regression model for each frame; and updating the target size in each frame of the trajectory segment of the target using the corrected target size for each frame.

[0112] For the target trajectory segment that triggers the correction condition, extract its temporal detection box sequence (including the 3D position, original size, and orientation angle of each frame). Map the adaptively pooled local BEV features of each frame to the temporal frames of the trajectory segment, forming a temporal feature tensor of [trajectory length × feature dimension]. Concatenate the detection results of the trajectory segment (such as initial size values ​​l, w, h) with the corresponding BEV feature tensor to construct the input data for the GRM model, ensuring that the temporal features are synchronized with the target state information.

[0113] The GRM model performs inference analysis, outputting the corrected dimensions (length l', width w', height h') for each frame. The corrected dimensions l', w', and h' output by the GRM are then used to replace the initial detection dimensions in the original trajectory segments frame by frame, outputting new trajectory segments with optimized dimensions while maintaining the position and orientation angle. These optimized dimensions are then output as the final tracking result to downstream tasks (such as path planning and behavior prediction). The corrected dimensions must temporally satisfy geometric consistency (e.g., the rate of change of dimensions between adjacent frames is less than a threshold) to avoid visual jitter caused by abrupt changes.

[0114] This application's embodiments refine the target trajectory size using a GRM model, significantly improving the geometric consistency and scene adaptability of multi-target tracking. The GRM model leverages global semantic information and local geometric details from BEV features, combined with temporal modeling capabilities, to accurately predict target size change trends, effectively eliminating size fluctuations caused by detection errors, sparse point clouds, or visual occlusion. By dynamically replacing the initial size in trajectory segments, it ensures the stability of target attributes during long-term tracking, reducing the risk of ID switching and trajectory breakage.

[0115] In some embodiments of this application, the geometric regression model is trained as follows: training sample data for the geometric regression model is constructed, the training sample data including a trajectory segment of the target, the corresponding BEV feature sequence, and the target's ground truth size sequence; the BEV feature sequence is cropped to obtain a local BEV feature sequence; the local BEV feature sequence is adaptively pooled to obtain an adaptively pooled local BEV feature sequence; the trajectory segment of the target, the adaptively pooled local BEV feature sequence, and the target's ground truth size sequence are input into the geometric regression model to obtain a size regression offset sequence predicted by the geometric regression model; the total loss value of the geometric regression model is calculated based on the size regression offset sequence, and the parameters of the geometric regression model are updated using the total loss value to obtain a trained geometric regression model.

[0116] When training the GRM model, training sample data needs to be constructed first. Based on the BEVFormer detection results and the Kalman filter tracking algorithm, initial trajectory segments (tracklets) containing temporally associated detection boxes are generated. Each segment records the target's 3D position, initial detection size, and orientation angle in consecutive frames.

[0117] For each frame of image covered by the trajectory segment, a global BEV feature map is generated using BEVFormer and associated with the trajectory data of the corresponding frame to form a BEV feature sequence F aligned with the trajectory time sequence. t By using high-precision sensor data (such as lidar point cloud fitting) or manual annotation, the true dimensions (length, width, and height) of the target in each frame within the trajectory segment are obtained, and a true dimension sequence is constructed as a supervision signal.

[0118] For each detection box in the trajectory segment, a cropping operation is performed on the BEV feature map, cropping local feature regions according to the pixel boundaries of the detection boxes to ensure precise alignment between features and target spatial positions. Then, an adaptive average pooling technique is used to unify the cropped multi-size local features to a fixed resolution, eliminating the influence of different target sizes on the model input while preserving the semantic integrity of the features.

[0119] The adaptively pooled local BEV feature sequence, the initial detection size of the trajectory segment, and the ground truth size sequence are integrated and input into the GRM model. The GRM model uses a temporal network (such as LSTM or Transformer encoder) as its core, takes the feature sequence and initial size as input, and outputs the size regression offset Δb for each frame. k,j This refers to the adjustment amount relative to the initial detection size.

[0120] The total loss of the model is calculated using a predefined loss function, such as temporal smoothing L1 loss. The difference between the predicted offset sequence and the true size sequence is compared, and a regularization term is introduced to prevent overfitting. Based on the total loss value, the GRM network parameters are updated through backpropagation, and the optimizer adopts an adaptive learning rate strategy (such as Adam).

[0121] Monitor the loss decrease trend on the validation set. When the loss change is less than the threshold for several consecutive epochs or the preset maximum number of iterations is reached, terminate training and save the optimal model parameters.

[0122] This application's embodiments achieve high-precision geometric consistency optimization capabilities by constructing training samples containing three elements: trajectory, features, and ground truth, combined with adaptive feature processing and temporal regression modeling. During training, the GRM model fully learns the spatial semantic information and temporal patterns of target size changes in BEV features, accurately predicting size regression offsets and effectively correcting size fluctuations caused by viewpoint changes, occlusion, or noise during initial detection. This solution, based on pure visual perception, eliminates the need for multi-sensor fusion, significantly reducing system complexity and deployment costs. It also exhibits strong robustness in complex traffic scenarios (such as overlapping targets and sparse point clouds), providing reliable target tracking correction support for autonomous driving systems and comprehensively improving the accuracy and reliability of the perception module.

[0123] This application embodiment also provides a target tracking device 200, such as Figure 2 The diagram shows a schematic representation of a target tracking device according to an embodiment of this application. The target tracking device 200 includes: an acquisition unit 210, a target detection unit 220, a target tracking unit 230, and a correction unit 240, wherein:

[0124] The acquisition unit 210 is used to acquire the multi-view image of the current frame captured by the multi-view camera;

[0125] The target detection unit 220 is used to perform target detection using a BEV detection model based on the multi-view image of the current frame, and obtain the 3D target detection result and the corresponding BEV features of the current frame.

[0126] The target tracking unit 230 is used to perform temporal tracking using a target tracking algorithm based on the 3D target detection result of the current frame to obtain an initial target tracking result;

[0127] The correction unit 240 is used to correct the initial target tracking result based on the initial target tracking result and the BEV feature using a geometric regression model to obtain the corrected target tracking result.

[0128] In some embodiments of this application, the target tracking unit 230 is specifically used to: use the target tracking algorithm to match the 3D target detection result of the current frame with the existing target trajectory segment to obtain a matching result; and based on the matching result, use the 3D target detection result of the current frame to update the existing target trajectory segment or create a new target trajectory segment.

[0129] In some embodiments of this application, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The correction unit 240 is specifically used to: determine whether the current frame triggers a preset target tracking correction condition based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame; if the preset target tracking correction condition is triggered, correct the initial target tracking result using the geometric regression model based on the initial target tracking result and the BEV feature to obtain the corrected target tracking result.

[0130] In some embodiments of this application, the correction unit 240 is specifically used to: determine whether the length of the trajectory segment of the target reaches a preset length threshold or a preset sliding window size based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame; if so, determine that the current frame triggers the preset target tracking correction condition; otherwise, determine that the current frame does not trigger the preset target tracking correction condition.

[0131] In some embodiments of this application, the initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The correction unit 240 is specifically used to: crop the BEV features of each frame corresponding to the trajectory segment of the target to obtain local BEV features of each frame; perform adaptive pooling on the local BEV features of each frame to obtain local BEV features of each frame after adaptive pooling; and correct the trajectory segment of the target using the geometric regression model based on the trajectory segment of the target and the local BEV features of each frame after adaptive pooling to obtain the corrected target tracking result.

[0132] In some embodiments of this application, the correction unit 240 is specifically used to: input the trajectory segment of the target and the local BEV features of each frame after adaptive pooling into the geometric regression model to obtain the corrected target size of each frame predicted by the geometric regression model; and update the target size of each frame in the trajectory segment of the target using the corrected target size of each frame.

[0133] In some embodiments of this application, the geometric regression model is trained as follows: training sample data for the geometric regression model is constructed, the training sample data including a trajectory segment of the target, the corresponding BEV feature sequence, and the target's ground truth size sequence; the BEV feature sequence is cropped to obtain a local BEV feature sequence; the local BEV feature sequence is adaptively pooled to obtain an adaptively pooled local BEV feature sequence; the trajectory segment of the target, the adaptively pooled local BEV feature sequence, and the target's ground truth size sequence are input into the geometric regression model to obtain a size regression offset sequence predicted by the geometric regression model; the total loss value of the geometric regression model is calculated based on the size regression offset sequence, and the parameters of the geometric regression model are updated using the total loss value to obtain a trained geometric regression model.

[0134] It is understood that the target tracking device described above can implement each step of the target tracking method provided in the foregoing embodiments. The relevant explanations of the target tracking method are applicable to the target tracking device and will not be repeated here.

[0135] Figure 3 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 3 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0136] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0137] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0138] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0139] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0140] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0141] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0142] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0143] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0144] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0145] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0146] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0147] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0148] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target tracking method, characterized in that, The target tracking method includes: Acquire multi-view images of the current frame captured by a multi-view camera; Based on the multi-view image of the current frame, target detection is performed using the BEV detection model to obtain the 3D target detection result and the corresponding BEV features of the current frame. Based on the 3D target detection results of the current frame, a target tracking algorithm is used for temporal tracking to obtain the initial target tracking results; Based on the initial target tracking results and the BEV features, the initial target tracking results are corrected using a geometric regression model to obtain the corrected target tracking results.

2. The target tracking method according to claim 1, characterized in that, The step of performing temporal tracking using a target tracking algorithm based on the 3D target detection results of the current frame to obtain the initial target tracking results includes: Using the target tracking algorithm, the 3D target detection result of the current frame is matched with the existing target trajectory fragments to obtain the matching result; Based on the matching results, the trajectory segments of existing targets are updated or new target trajectory segments are created using the 3D target detection results of the current frame.

3. The target tracking method according to claim 1, characterized in that, The initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The initial target tracking result is then corrected using a geometric regression model based on the initial target tracking result and the BEV features, resulting in the corrected target tracking result, which includes: Determine whether the preset target tracking correction condition is triggered in the current frame based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame; When the preset target tracking correction condition is triggered, the initial target tracking result is corrected using the geometric regression model based on the initial target tracking result and the BEV feature, to obtain the corrected target tracking result.

4. The target tracking method according to claim 3, characterized in that, The step of determining whether the preset target tracking correction condition is triggered in the current frame based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame includes: Based on the trajectory segment of the target corresponding to the 3D target detection result of the current frame, determine whether the length of the target trajectory segment reaches a preset length threshold or a preset sliding window size; If so, then the current frame is determined to trigger the preset target tracking correction condition; Otherwise, it is determined that the current frame has not triggered the preset target tracking correction condition.

5. The target tracking method according to claim 1, characterized in that, The initial target tracking result includes the trajectory segment of the target corresponding to the 3D target detection result of the current frame. The initial target tracking result is then corrected using a geometric regression model based on the initial target tracking result and the BEV features, resulting in the corrected target tracking result, which includes: The BEV features of each frame corresponding to the trajectory segment of the target are cropped to obtain the local BEV features of each frame. Adaptive pooling is performed on the local BEV features of each frame to obtain the adaptively pooled local BEV features of each frame. Based on the trajectory segment of the target and the local BEV features of each frame after adaptive pooling, the trajectory segment of the target is corrected using the geometric regression model to obtain the corrected target tracking result.

6. The target tracking method according to claim 5, characterized in that, The step of correcting the target's trajectory segment using the geometric regression model based on the target's trajectory segment and the adaptively pooled local BEV features of each frame to obtain the corrected target tracking result includes: The trajectory fragment of the target and the local BEV features of each frame after adaptive pooling are input into the geometric regression model to obtain the corrected target size predicted by the geometric regression model for each frame. The target size in each frame of the trajectory segment of the target is updated using the corrected target size for each frame.

7. The target tracking method according to any one of claims 1 to 6, characterized in that, The geometric regression model is trained in the following manner: The training sample data for constructing the geometric regression model includes trajectory segments of the target, corresponding BEV feature sequences, and ground truth size sequences of the target. The BEV feature sequence is cropped to obtain a local BEV feature sequence; The local BEV feature sequence is adaptively pooled to obtain the adaptively pooled local BEV feature sequence. The trajectory segment of the target, the adaptively pooled local BEV feature sequence, and the true size sequence of the target are input into the geometric regression model to obtain the size regression offset sequence predicted by the geometric regression model. The total loss value of the geometric regression model is calculated based on the size regression offset sequence, and the parameters of the geometric regression model are updated using the total loss value to obtain the trained geometric regression model.

8. A target tracking device, characterized in that, The target tracking device includes: The acquisition unit is used to acquire the multi-view image of the current frame captured by the multi-view camera; The target detection unit is used to perform target detection using the BEV detection model based on the multi-view image of the current frame, and obtain the 3D target detection result and the corresponding BEV features of the current frame. The target tracking unit is used to perform temporal tracking using a target tracking algorithm based on the 3D target detection results of the current frame to obtain an initial target tracking result; The correction unit is used to correct the initial target tracking result based on the initial target tracking result and the BEV feature using a geometric regression model to obtain the corrected target tracking result.

9. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the target tracking methods of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the target tracking method according to any one of claims 1 to 7.

Images