Multi-target joint detection and tracking method based on timing representation enhancement and trajectory correction

Abstract

The application relates to a multi-target joint detection and tracking method based on time sequence characteristic enhancement and trajectory correction, which comprises the following steps: in a multi-target joint detection and tracking process, panoramic images and point clouds are collected, feature extraction is respectively performed, the extracted features are fused, and multi-modal fusion features are obtained; a time sequence characteristic of a target query vector initialized by a current frame is enhanced by using a historical trajectory query vector, spatial feature interaction is performed between the multi-modal fusion features and the target query vector, and then target decoding is performed to obtain a target detection result; data correlation is performed according to the detection result, whether target identity inheritance is performed is judged, and a trajectory ID is obtained; the historical query vector is updated based on a target trajectory tracked in the current frame, trajectory prediction is performed based on the historical query vector and map information; and low-confidence trajectories are corrected by using the predicted target trajectory according to the classification confidence of the target detection result. Compared with the prior art, the application improves the accuracy and robustness of target detection and tracking.

Description

Technical Field

[0001] This invention relates to the field of environmental perception for autonomous driving, and in particular to a multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction. Background Technology

[0002] In recent years, with the rapid development of new energy vehicles, intelligent driving technology has also become widespread. Environmental perception technology, as one of the three core technologies of intelligent driving systems, is used to perceive the vehicle's status and the surrounding environment.

[0003] For dynamic targets in road scenarios, it is necessary not only to accurately detect their current position but also to effectively track their trajectory. The basic process of multi-target tracking includes four parts: target localization, feature extraction, similarity measurement, and data association. Target localization is achieved through target detection, feature extraction is used for similarity measurement, and data association aims to match detected targets with their corresponding historical trajectories based on the similarity between targets to maintain target identity. Multi-target tracking methods mainly include detection-based tracking paradigms and joint detection-tracking paradigms. Traditional detection-based tracking methods perform target detection first, followed by multi-target tracking; this method is computationally fast but cannot perform joint optimization. Joint detection-tracking methods can achieve joint optimization and are an end-to-end perception algorithm.

[0004] The appearance of a target may change during movement, which poses a significant challenge to the robustness of the joint detection and tracking model. Target occlusion or other detection noise can significantly reduce the target confidence level, causing changes in the target trajectory ID and leading to a decrease in the accuracy and robustness of the joint detection and tracking model. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction. Temporal representation enhancement makes target representation more effective and identification more accurate, while trajectory correction reduces trajectory ID changes caused by decreased target confidence and makes the trajectory more coherent, thereby improving the accuracy and robustness of detection and tracking.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction includes the following steps:

[0008] During target tracking, a panoramic image and point cloud are acquired, and features are extracted from both the panoramic image and the point cloud. The extracted features are then fused to obtain multimodal fusion features. The target query vector initialized in the current frame is enhanced with temporal representation using historical trajectory query vectors, and spatial features are interacted with the multimodal fusion features. After target decoding, the target detection result is obtained.

[0009] Based on the target detection results, data association is performed to determine whether target identity inheritance should be performed, and the trajectory ID is obtained.

[0010] Based on the target trajectory tracked in the current frame, update the pre-stored historical query vector, and perform trajectory prediction based on the historical query vector and map information;

[0011] Based on the target classification confidence of the target detection results, the low-confidence trajectory is corrected using the predicted target trajectory.

[0012] Furthermore, the process of feature extraction from the panoramic image is as follows:

[0013] The panoramic image is input into a pre-built feature extraction network to obtain image features. The 3D spatial coordinates of the panoramic image are projected onto a higher-dimensional feature space using a position embedding module to obtain 3D position feature encoding. 3D positional features are encoded through a fusion module. Integrating into image features In this process, 3D perceived image features are obtained.

[0014] Furthermore, the process of feature extraction from point clouds is as follows:

[0015] By using a pre-built point cloud feature extraction network, feature extraction is performed on the point cloud data of the current frame to obtain point cloud features.

[0016] Furthermore, the process of enhancing the temporal representation specifically includes:

[0017] The historical trajectory query vector is aligned to the current frame through motion compensation and motion feature embedding, and then features are interacted with the query vector of the current frame based on a temporal attention mechanism. The calculation expression for this feature interaction is as follows:

[0018]

[0019] In the formula, The feature query vector is obtained after temporal representation enhancement, where TA(·) is the temporal attention module; motion guidance is achieved through motion feature embedding; the query vector is defined based on the Transformer attention architecture, and the query vector Q is derived from the feature query vector Q. c and location query vector Qp composition; and These are the feature query vector and the position embedding query vector input to the i-th layer of the L-layer Transformer model before temporal augmentation, respectively; when i = 0... This is the initial query vector for the current frame, and serves as the initial input to the temporal attention module. and These are the historical feature query vector and the historical location query vector, respectively, and their corresponding calculation expressions are:

[0020]

[0021] In the formula, cct(·) is the concatenation operation. and They are k h The location embedding query vector and feature query vector of each historical frame.

[0022] Furthermore, the motion compensation specifically involves: transforming the historical position query vector to the current frame using the vehicle's posture, and then embedding the position. The corresponding calculation expression is:

[0023]

[0024] In the formula, Ψ(·) is the motion compensation operator; It is a historical location query vector. The pose of the vehicle in the historical frame is used as the basis for motion compensation, which yields the pose transformation matrix from the historical frame to the current frame. and historical location query vector ε(·) is the location embedding module, aligned to the historical location query vector of the current frame. Then, perform location embedding to obtain the historical location embedding query vector. Historical feature query vector It remains unchanged when aligned to the current frame;

[0025] The motion feature embedding specifically refers to:

[0026] The pose transformation matrix, inter-frame time difference, and target prediction velocity are embedded into the query vector through affine transformation during the layer normalization process. The corresponding calculation expression is:

[0027]

[0028] In the formula, LN(·) represents the layer normalization operator without learnable affine transformation parameters, and the calculation expressions for parameters α and β are as follows:

[0029]

[0030] In the formula, M is the motion feature, which includes the product of the inter-frame time difference and the target prediction velocity, and the transformation matrix from the historical frame to the current frame. The motion feature M is first transformed into a motion feature vector through the same linear layer L0, and then the affine transformation coefficients α and β with motion features are obtained through different linear layers L1 and L2 respectively.

[0031] Furthermore, the current frame initial query vector Query vector from the previous frame aligned to the current frame and the initial query vector It is composed of two parts, and the corresponding calculation expression is:

[0032]

[0033] In the formula, cct(·) is the concatenation operation; From the initialized position query vector Position embedding is performed to obtain, It is a feature query vector consisting of all zeros with the same shape as the location query vector, and its corresponding calculation expression is:

[0034]

[0035] In the formula, ε(·) is the position embedding module, and zero_like(·) is the operator that generates a vector of all zeros of the same shape.

[0036] Furthermore, the calculation expression for the spatial feature interaction is:

[0037]

[0038] In the formula, SA(·) is the feature query vector obtained after spatial feature interaction; SA(·) is the spatial attention module. It is the feature query vector output by the Lth layer of the Transformer model, which is a temporal representation enhancement module; the positional embedding query vector remains unchanged in different Transformer layers, i.e. It is the multimodal fusion feature of the current frame.

[0039] Furthermore, the data association employs different association strategies to obtain the target ID during the model training and inference phases;

[0040] The expression for the association strategy during the model training phase is as follows:

[0041]

[0042] In the formula, for a target with an ID in the current frame, if the i-th target at time t-1... In the current frame truth value ID set In the current frame, the target inherits the ID from the previous frame and is marked as an active "act". If the target ID is not in the set of truth IDs for the current frame... If the condition is met, then determine the number of consecutive frames that are not in the set of truth IDs. With frame number threshold l X If the value is less than the threshold, then the target ID from the previous frame is inherited. The target is marked as deact (inactive state). If the value is greater than the threshold, the target is considered to have left the field of view and the corresponding trajectory is deleted. For targets without identity information in the current frame, the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box that did not appear in the previous frame is used to determine whether it is a real target. If the IoU is greater than the threshold, the ground truth ID is assigned to the predicted bounding box and its state is set to act (active state). If the IoU is less than the threshold, it is considered a false positive (FP).

[0043] The expression for the association strategy in the model inference phase is:

[0044]

[0045] In the formula, if the classification confidence s corresponds to the trajectory query vector, then the decoded classification confidence is... Greater than threshold ∈ trk The time frame is considered the real target, inheriting the trajectory from the previous frame. Less than threshold ∈ trk If the classification confidence s corresponds to the initialized query vector, then the decoded classification confidence is considered a false positive (FP). Greater than threshold ∈ init If the value is less than the threshold ε, it is considered a real target and a new ID is assigned. init This is considered a false positive (FP).

[0046] Furthermore, the trajectory prediction process specifically involves: predicting the trajectory of future kf frames based on stored historical query vectors using a prediction module, with the corresponding calculation expression being:

[0047]

[0048] In the formula, Γ(·) is the map information encoding module, and I map It is map information. First, the map information is encoded using a map information encoding module to obtain map encoding information. Then, historical feature query vectors are used. and map coding features F m Predicting future κ using the time attention module TA(·) f Frame query vector Finally, the future κ is obtained by decoding using the multilayer sensing module MLP(·). fFrame target motion

[0049] Furthermore, the calculation expression for correcting the low-confidence trajectory using the predicted target trajectory is as follows:

[0050]

[0051] In the formula, whether trajectory correction is needed is determined based on the target classification confidence level predicted by the model. If the target classification confidence level is high... Greater than threshold ∈ r Then, the predicted trajectory is used to query the vector of the current frame. and Passed to the next frame and Using the trajectory extension operator Obtain the t+κth f +1 frame of motion, and continuously correct the target trajectory for the number of frames. Zero; if the target classification confidence level is zero. Less than threshold ∈ r First, determine if the number of consecutive corrected frames for the target trajectory is less than the threshold ∈ L If the target trajectory is continuously corrected for a number of frames Less than threshold ∈ L If the target has temporarily left the field of view, the trajectory is retained, and the target query vector is passed to the next frame using the extended trajectory from the previous frame, and the trajectory extension operator is used. Obtain the t+κth f +1 frame of motion, if the target trajectory is continuously corrected for the number of frames. Greater than threshold ∈ L If the target has left the field of view, the corresponding trajectory will be deleted.

[0052] Furthermore, during the model training process, the detection part of the model is first trained in a single frame, and then the entire model is trained in a multi-frame joint process. The model training process includes data augmentation of the model input. During single-frame training, the data augmentation includes scaling, cropping, flipping, rotating, and translating. During multi-frame training, the data augmentation includes scaling and cropping.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] (1) The present invention includes the following steps: In the process of multi-target joint detection and tracking, a panoramic image and a point cloud are collected, features are extracted from the panoramic image and the point cloud respectively, and the extracted features are fused to obtain multimodal fusion features; the target query vector initialized in the current frame is enhanced with the historical trajectory query vector and interacted with the multimodal fusion features in terms of spatial features, and then the target is decoded to obtain the target detection result; data association is performed according to the target detection result, and it is determined whether to perform target identity inheritance to obtain the trajectory ID; based on the target trajectory tracked in the current frame, the pre-stored historical query vector is updated, and trajectory prediction is performed based on the historical query vector and map information; based on the target classification confidence of the target detection result, the low confidence trajectory is corrected using the predicted target trajectory.

[0055] This scheme enhances target representation and identification by performing temporal representation enhancement on multimodal fusion features. It also uses predicted trajectories to correct low-confidence trajectories, reducing trajectory ID changes and making trajectories more coherent, thereby improving the accuracy and robustness of target detection and tracking.

[0056] (2) The present invention first performs single-frame training on the detection part of the model, and then performs multi-frame joint training on the entire model. This training method is conducive to model convergence and can speed up the model training speed.

[0057] (3) When training the model, the present invention performs various augmentations on the input data. In the single-frame training detection part, augmentation methods such as scaling, cropping, flipping, rotating and translating are used. In the multi-frame joint training, augmentation methods such as scaling and cropping are used to increase the diversity of input data and enhance the accuracy and robustness of the model. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating a multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction provided in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the model structure of a multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction provided in an embodiment of the present invention;

[0060] Figure 3 This is a flowchart illustrating a motion-guided time alignment method provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0062] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0063] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0064] Example 1

[0065] This embodiment provides an end-to-end multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction, including the following steps:

[0066] S1: During multi-target tracking, a panoramic image and point cloud are acquired. Features are extracted from the panoramic image and point cloud respectively. The extracted features are fused to obtain multimodal fusion features. The target query vector initialized in the current frame is enhanced with temporal representation using the historical trajectory query vector. Spatial feature interaction is performed with the multimodal fusion features. Then, target decoding is performed to obtain the target detection result.

[0067] S2: Based on the target detection results, perform data association to determine whether to perform target identity inheritance and obtain the trajectory ID;

[0068] S3: Based on the target trajectory tracked in the current frame, update the pre-stored historical query vector, and perform trajectory prediction based on the historical query vector and map information;

[0069] S4: Based on the target classification confidence of the target detection results, use the predicted target trajectory to correct the low-confidence trajectory.

[0070] Specifically, in step S1, feature extraction and multimodal feature fusion are first performed on the panoramic image and point cloud, then motion-guided temporal representation enhancement and spatial feature interaction are performed, and finally target decoding is performed to obtain the prediction box, category and confidence.

[0071] First, the current frame's circular view image is processed by a feature extraction network, and then integrated with 3D location embedding to obtain 3D-aware image features, i.e.

[0072]

[0073] In the formula, It is a feature extraction network, including a backbone network and a feature fusion network. It extracts the surrounding image. Image features are obtained by inputting the feature extraction network. The position embedding module, ε(·), then projects the 3D spatial coordinates onto a higher-dimensional feature space to obtain the 3D position feature encoding. Then, the 3D position is embedded into the image features through the fusion module φ(·) to obtain the 3D perceived image features.

[0074] The point cloud of the current frame is processed by a feature extraction network to obtain point cloud features:

[0075]

[0076] In the formula, It is a point cloud feature extraction network, P t It is the point cloud data of the current frame.

[0077] As an optional implementation, the point cloud feature extraction network is the SECOND model.

[0078] 3D perceived image features and point cloud features are used to obtain fused features through a multimodal feature fusion module:

[0079]

[0080] In the formula, Θ(·) is a multimodal feature fusion network. It is the obtained multimodal fusion feature of the current frame.

[0081] Then, the historical trajectory query vector is aligned to the current frame through motion compensation and motion feature embedding, and then features are interacted with the current frame query vector based on a time attention mechanism:

[0082]

[0083] In the formula, The query vector is the feature query vector obtained after temporal representation enhancement, where TA(·) is the temporal attention module; the query vector is defined based on the Transformer attention architecture, and the query vector Q is derived from the feature query vector Q. c and location query vector Q p composition; and These are the feature query vector and the position embedding query vector input to the i-th layer of the L-layer Transformer model before temporal augmentation; when i = 0... This is the initial query vector for the current frame, and serves as the initial input to the temporal attention module. and These are the historical feature query vector and the historical location query vector, respectively, obtained by the following formulas:

[0084]

[0085] In the formula, cct(·) is the concatenation operation. and They are κ h The location embedding query vector and feature query vector of each historical frame.

[0086] As an optional implementation, the number of Transformer model layers L is 6.

[0087] As an optional implementation, motion compensation involves transforming the historical position query vector in the storage module to the current frame based on the vehicle's posture, and then embedding the position.

[0088]

[0089] In the formula, Ψ(·) is the motion compensation operator; It is a historical location query vector. The pose of the vehicle in the historical frame is used as the basis for motion compensation, which yields the pose transformation matrix from the historical frame to the current frame. Historical location query vector ε(·) is the position embedding module, which aligns to the historical position query vector of the current frame and then performs position embedding to obtain the historical position embedding query vector. Historical feature query vector It remains unchanged when aligned to the current frame.

[0090] As an optional implementation method, such as Figure 3 As shown, motion feature embedding involves embedding the pose transformation matrix, inter-frame time difference, and target predicted velocity into the query vector through affine transformation during the layer normalization process, i.e.:

[0091]

[0092] In the formula, LN(·) denotes the layer normalization operator without learnable affine transformation parameters; the parameters α and β are obtained from the following formula:

[0093]

[0094] In the formula, M is the motion feature, which includes the product of the inter-frame time difference and the target prediction velocity, and the transformation matrix from the historical frame to the current frame. The motion feature M is first transformed into a motion feature vector through the same linear layer L0, and then the affine transformation coefficients α and β with motion features are obtained through different linear layers L1 and L2 respectively.

[0095] As an optional implementation, the initial query vector of the current frame Query vector from the previous frame aligned to the current frame and the initial query vector It is pieced together, that is

[0096]

[0097] In the formula, cct(·) is the concatenation operation; From the initialized position query vector Position embedding is performed to obtain, It is a feature query vector consisting of all zeros and having the same shape as the location query vector, i.e.:

[0098]

[0099] In the formula, ε(·) is the position embedding module, and zero_like(·) is the operator that generates a vector of all zeros of the same shape.

[0100] Then, the feature query vector with enhanced temporal representation. Further fusion features with the current frame's multimodal features Perform spatial feature interaction:

[0101]

[0102] In the formula, SA(·) is the spatial attention module; It is the feature query vector output by the Lth layer of the Transformer model, which is a temporal representation enhancement module; the positional embedding query vector remains unchanged in different Transformer layers, i.e. It is the multimodal fusion feature of the current frame.

[0103] Finally, the feature query vector obtained after spatial feature interaction is used by the target decoding module to obtain the target's predicted bounding box, category, and confidence score.

[0104]

[0105] In the formula, It is the model's decoding module. It is the feature query vector output by the Lth layer Transformer of the spatial feature interaction module.

[0106] In step S2, the detected target is correlated with data to obtain the trajectory ID.

[0107] Specifically, different association strategies are used to obtain the target ID during the model training and inference phases.

[0108] As an optional implementation method, the data association strategy during the model training phase is as follows:

[0109]

[0110] In the formula, for a target with an ID in the current frame, if the i-th target at time t-1... In the current frame truth value ID set In the current frame, the target inherits the ID from the previous frame and is marked as an active "act". If the target ID is not in the set of truth IDs for the current frame... If the condition is met, then determine the number of consecutive frames that are not in the set of truth IDs. With frame number threshold l X If the value is less than the threshold, then the target ID from the previous frame is inherited. The target is marked as deact. If the value is greater than the threshold, the target is considered to have left the field of view and the trajectory is deleted. For targets without identity information in the current frame, the intersection-over-union ratio (IoU) between the predicted bounding box and the ground truth bounding box that did not appear in the previous frame is used to determine whether it is a real target. If the IoU is greater than the threshold, the ground truth ID is assigned to the predicted bounding box and its state is set to act. If the IoU is less than the threshold, it is considered a false positive (FP).

[0111] As an optional implementation, the association strategy for data association during the model inference phase is as follows:

[0112]

[0113] In the formula, if the classification confidence s corresponds to the trajectory query vector, then the decoded classification confidence is... Greater than threshold ∈ trk The time frame is considered the real target, inheriting the trajectory from the previous frame. Less than threshold ∈ trk If the classification confidence s corresponds to the initialized query vector, then the decoded classification confidence is considered a false positive (FP). Greater than threshold ∈ init If the value is less than the threshold ε, it is considered a real target and a new ID is assigned. init This is considered a false positive (FP).

[0114] As an optional implementation, the classification confidence threshold ∈ trk and ∈ init Take values ​​of 0.2 and 0.3 respectively.

[0115] In step S3, the storage module is updated first, and then trajectory prediction is performed based on historical query vectors and combined with map information.

[0116] Specifically, firstly, the predicted targets are sorted based on the predicted class confidence scores, the top k targets are selected, and their corresponding query vectors are added to the storage module. At the same time, all target query vectors from the earliest frame in the storage module are deleted.

[0117]

[0118] As an optional implementation, the storage module uses a first-in-first-out strategy to store historical information, which includes N historical frames and k trajectory query vectors for each frame.

[0119] As an optional implementation, the first 256 targets are selected for storage, the number of historical frames N is 4, and the number of trajectory query vectors k in each frame is 256.

[0120] Then, based on the historical query vectors in the storage module, the prediction module predicts the future κ. f Predicting the trajectory of a frame:

[0121]

[0122] In the formula, Γ(·) is the map information encoding module, and I map It is map information. First, the map information is encoded using a map information encoding module to obtain map encoding information. Then, historical feature query vectors are used. and map coding features F m Predicting future κ using the time attention module TA(·) f Frame query vector Finally, the future κ is obtained by decoding using the multilayer sensing module MLP(·). f Frame target motion

[0123] In step S4, the predicted trajectory is used to correct the low-confidence trajectory.

[0124] Specifically, firstly, the confidence level of the target classification predicted by the model is used to determine whether trajectory correction is needed:

[0125]

[0126] If the target classification confidence level Greater than threshold ∈ r In this case, trajectory correction is not required; the predicted trajectory can be used to query the vector of the current frame. and Passed to the next frame and Using the trajectory extension operator Obtain the t+kth digit f +1 frame of motion, and continuously correct the target trajectory for the number of frames. Reset to zero. If the target classification confidence level is zero. Less than threshold ∈ r If so, trajectory correction is required. First, determine if the number of consecutive frames for target trajectory correction is less than a threshold ∈ L If the target trajectory is continuously corrected for a number of frames Less than threshold ∈ L If the target has temporarily left the field of view, the trajectory is retained, and the target query vector is passed to the next frame using the extended trajectory from the previous frame, replacing the low-confidence trajectory in the current frame, and the trajectory extension operator is used. Obtain the t+kth digit f +1 frame of motion; if the target trajectory is continuously corrected for the number of frames. Greater than threshold ∈ L If the target has left the field of vision, the trajectory will be deleted.

[0127] As an optional implementation, the classification confidence threshold ∈ r and continuous correction frame number threshold ∈ L They are 0.2 and 8 respectively.

[0128] As an optional implementation, trajectory prediction for future frames κ f The value is 8.

[0129] As an optional implementation, the model training process includes first training the detection part of the model on a single frame, and then jointly training the entire model across multiple frames. Joint training of the detection and tracking model requires at least two frames of data, preferably three. Joint optimization achieves mutual enhancement between the detection and tracking parts, improving the model's detection and tracking accuracy.

[0130] As an optional implementation method, the loss function used in the model training process is:

[0131]

[0132] In the formula, Focal loss is used to supervise the classification of targets in the target detection module, and its coefficient is... The value is 1.0; The L1 loss is used to supervise the regression of the predicted bounding boxes in the object detection module, and its coefficient is... The value is 1.0; The L1 loss is used to supervise trajectory prediction in the trajectory prediction module, and its coefficient is... The value is 0.2.

[0133] As an optional implementation, the model training process includes data augmentation of the model input to increase the diversity of the input data and improve the robustness of the model.

[0134] As an optional implementation, different augmentation methods are used for single-frame and multi-frame training, including scaling, cropping, flipping, rotating, and translating for single-frame training, and scaling and cropping for multi-frame training.

Claims

1. A multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction, characterized in that, Includes the following steps: During target tracking, a panoramic image and point cloud are acquired, and features are extracted from both the panoramic image and the point cloud. The extracted features are then fused to obtain multimodal fusion features. The target query vector initialized in the current frame is enhanced with temporal representation using historical trajectory query vectors, and spatial features are interacted with the multimodal fusion features. After target decoding, the target detection result is obtained. Based on the target detection results, data association is performed to determine whether target identity inheritance should be performed, and the trajectory ID is obtained. Based on the target trajectory tracked in the current frame, update the pre-stored historical query vector, and perform trajectory prediction based on the historical query vector and map information; Based on the target classification confidence score of the target detection results, the low-confidence trajectory is corrected using the predicted target trajectory; The process of enhancing the temporal representation is specifically as follows: The historical trajectory query vector is aligned to the current frame through motion compensation and motion feature embedding, and then features are interacted with the query vector of the current frame based on a temporal attention mechanism. The calculation expression for this feature interaction is as follows: In the formula, This is the feature query vector obtained after time-series representation enhancement. It is a temporal attention module; motion guidance is achieved through motion feature embedding; the query vector is based on... Definition of attention architecture, query vector From feature query vector and location query vector composition; and They are layer Model No. The layer inputs the feature query vector and the position embedding query vector before temporal augmentation; when hour, This is the initial query vector for the current frame, and serves as the initial input to the temporal attention module. and These are the historical feature query vector and the historical location query vector, respectively, and their corresponding calculation expressions are: In the formula, It's a splicing operation. and They are The location embedding query vector and feature query vector of each historical frame.

2. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The specific process of feature extraction from a panoramic image is as follows: The panoramic image is input into a pre-built feature extraction network to obtain image features. ; The location embedding module corresponds to the surround view image. The spatial coordinates are projected onto a higher-dimensional feature space to obtain Location feature encoding Through the fusion module Location feature encoding Integrating into image features In the middle, we obtained Perceived image features.

3. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The motion compensation specifically refers to: The historical location query vector is transformed to the current frame by the vehicle's attitude, and then the location is embedded. The corresponding calculation expression is: In the formula, It is a motion compensation operator; It is a historical location query vector. The pose of the vehicle in the historical frame is used as the basis for motion compensation, which yields the pose transformation matrix from the historical frame to the current frame. and historical location query vector ; It is a location embedding module that aligns to the historical location query vector of the current frame. Then, perform location embedding to obtain the historical location embedding query vector. ; Historical feature query vector It remains unchanged when aligned to the current frame; The motion feature embedding specifically refers to: The pose transformation matrix, inter-frame time difference, and target prediction velocity are embedded into the query vector through affine transformation during the layer normalization process. The corresponding calculation expression is: In the formula, This represents a layer normalization operator without learnable affine transformation parameters, where the parameters are... and The calculation expression is: In the formula, These are motion features, including the product of inter-frame time difference and target predicted velocity, and the transformation matrix from historical frames to the current frame; motion features First through the same linear layer Transformed into motion feature vectors, and then passed through different linear layers. , Obtain affine transformation coefficients with motion characteristics and .

4. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 2, characterized in that, The current frame initial query vector Query vector from the previous frame aligned to the current frame and the initial query vector It is composed of two parts, and the corresponding calculation expression is: In the formula, It's a splicing operation; From the initialized position query vector Position embedding is performed to obtain, It is a feature query vector consisting of all zeros with the same shape as the location query vector, and its corresponding calculation expression is: In the formula, It is a location embedding module. It is an operator that generates all-zero vectors of the same shape.

5. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The calculation expression for the spatial feature interaction is: In the formula, This is the feature query vector obtained after spatial feature interaction; It is a spatial attention module; It is a time-series representation enhancement module Model No. Feature query vectors output by the layer; different The position embedding of the query vector within the layer remains unchanged, i.e. ; It is the multimodal fusion feature of the current frame.

6. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The data association employs different association strategies to obtain target numbers during the model training and inference phases. ; The expression for the association strategy during the model training phase is as follows: In the formula, for the current frame, there is If the goal is Time of the first One goal In the current frame truth value gather In the middle, the target of the current frame inherits the target of the previous frame. And mark the target as active. If the target Not in the current frame truth value gather If the condition is not true, then determine if the continuous value is not true. Number of frames in the collection With frame number threshold If the value is less than the threshold, the target from the previous frame is inherited. Right now Mark the target as inactive. If the value exceeds a threshold, the target is considered to have left the field of view, and the corresponding trajectory is deleted. For targets without identity information in the current frame, the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box that did not appear in the previous frame is used. Determine if it is a real target; if the intersection-union ratio (IU) is greater than a threshold, then set the true value. Assign it to the prediction box and set its state to active. If the crossover ratio is less than the threshold, it is considered a false alarm. ; The expression for the association strategy in the model inference phase is: In the formula, if the classification confidence level The corresponding trajectory query vector is the decoded classification confidence score. Greater than the threshold The time frame is considered the real target, inheriting the trajectory from the previous frame. less than the threshold This is considered a false alarm. If the classification confidence level Corresponding to the initialized query vector, the decoded classification confidence score is... Greater than the threshold Time is considered the real target, and new allocations are made. less than the threshold This is considered a false alarm. .

7. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The trajectory prediction process specifically involves: based on stored historical query vectors, the prediction module predicts future... The trajectory of the frame is predicted, and the corresponding calculation expression is: In the formula, It is the map information encoding module. It is map information. First, the map information is encoded using a map information encoding module to obtain map encoding information. Then, historical feature query vectors are used. and map coding features Through the time attention module Predicting the future Frame query vector Finally, the multi-layer sensing module is used. Decoding the Future Frame target motion .

8. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, The calculation expression for correcting low-confidence trajectories using predicted target trajectories is as follows: In the formula, whether trajectory correction is needed is determined based on the target classification confidence level predicted by the model. If the target classification confidence level is high... Greater than the threshold Then, the predicted trajectory is used to query the vector of the current frame. and Passed to the next frame and Using the trajectory extension operator Obtain the The motion of the frames, and the continuous correction of the target trajectory by the number of frames. Zero; if the target classification confidence level is zero. Less than the threshold First, determine whether the number of consecutive frames for correcting the target trajectory is less than the threshold. If the target trajectory is continuously corrected for a number of frames Less than the threshold If the target has temporarily left the field of view, the trajectory is retained, and the target query vector is passed to the next frame using the extended trajectory from the previous frame, and the trajectory extension operator is used. Obtain the The motion of the frame, if the target trajectory is continuously corrected for the number of frames. Greater than the threshold If the target has left the field of view, the corresponding trajectory will be deleted.

9. The multi-target joint detection and tracking method based on temporal representation enhancement and trajectory correction according to claim 1, characterized in that, During the model training process described in the method, the detection part of the model is first trained in a single frame, and then the entire model is trained in a multi-frame joint process. The model training process includes data augmentation of the model input. In single-frame training, the data augmentation includes scaling, cropping, flipping, rotating, and translating. In multi-frame training, the data augmentation includes scaling and cropping.

Images