A multi-target tracking method based on long- and short-term trajectory correlation

Abstract

This invention belongs to the field of target tracking technology and discloses a multi-target tracking method based on long-short-term trajectory association. Addressing the shortcomings of existing methods such as insufficient collaborative optimization of detection and tracking tasks in complex scenes, limited accuracy due to limited long-term trajectory modeling capabilities, and frequent trajectory breaks and identity switching, this invention proposes a multi-target tracking method based on long-short-term trajectory association: A YOLOX-based detection module generates target detection boxes and detection queries; a MOTR-ConvNext network dynamically updates the trajectory queries and predicts target motion information; a trajectory query memory module stores and updates historical trajectories; a matching module performs IOU matching between detection boxes and predicted boxes to achieve short-term association; a historical backtracking module calculates cross-frame correlations for unmatched detections to achieve long-term association; finally, the two association results are combined to obtain a complete tracking result. This invention can significantly improve tracking accuracy in complex scenes and also has stronger robustness in maintaining target identity.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, and in particular relates to a multi-target tracking method based on long- and short-term trajectory correlation. Background Technology

[0002] Multi-object tracking (MOT) is an important research direction in computer vision. However, in complex scenes, due to factors such as similar object appearances and long-term occlusion, problems such as trajectory breakage and identity switching often occur. Although the detection-by-detection paradigm has made significant progress by separating detection and association tasks, its reliance on heuristic matching rules (such as IoU and Re-ID features) and post-processing strategies (such as trajectory interpolation) makes it impossible to jointly optimize detection and association tasks. Furthermore, it is difficult to model long-term time series to address issues such as long-term occlusion, object disappearance, and identity switching in complex motion scenes.

[0003] In recent years, Transformer-based end-to-end methods have significantly improved the robustness of short-term correlations by implicitly modeling target motion and appearance changes through iterative track query updates. However, these methods still suffer from two major bottlenecks. First, there is an optimization conflict between detection and tracking tasks. Existing methods jointly optimize detection queries and track queries, but detection performance is limited by the Transformer end-to-end framework, leading to high false negative and false positive rates. While some improvements have attempted to alleviate the detection problem by introducing pre-trained detectors to generate proposal queries, the implicit decoupling of detection and tracking still cannot completely eliminate gradient interference between tasks. Second, there is insufficient modeling of long-term dependencies. Existing methods mainly rely on local information from adjacent frames, lacking a global awareness of the historical context of multiple frames. For example, some methods use temporal aggregation mechanisms to fuse trajectory features from the current frame and the previous frame to preserve short-term trajectory information, but their online processing mechanisms and exit strategies limit the ability to trace the trajectory of targets that have been occluded or disappeared for a long time. Other methods provide position priors through detectors, but lack a recovery mechanism for long-term lost targets, making it difficult to solve the trajectory breakage problem in occluded scenarios.

[0004] To address the above issues, a series of solutions have recently been proposed in this field through multi-dimensional innovation:

[0005] Patent CN116681729A proposes a two-level association matching and hypothesis box tracking method. This method generates target prediction boxes through trajectory prediction, initiates secondary matching based on neighbor graph relationships after the initial matching fails, and combines a hypothesis box continuous tracking mechanism to address the target loss problem under severe occlusion. This method has demonstrated its anti-occlusion capability on the MOT Challenge dataset, but it does not solve the joint optimization problem of detection and tracking.

[0006] Patent CN116758110A addresses complex motion scenarios by designing an IoU-ReID fusion matching and observation amplification method. This method improves the matching probability of occluded targets by adaptively weighting appearance and motion features and expanding the prediction box size. Its trajectory state management module further optimizes the stability of long-term tracking, but it still falls short in long-term modeling of nonlinear motion.

[0007] Patent CN117252908A proposes a multi-scale dual attention network that focuses on tracking target regions through spatial and channel attention, and designs a temporal aggregation network to fuse historical features to suppress occlusion noise. Its adaptive weight mechanism dynamically adjusts the feature enhancement intensity, effectively improving identity consistency in complex interaction scenarios, but the recovery of targets that have disappeared for a long time still relies on an external detector.

[0008] Patent CN119048563A proposes a confidence-based selective feature update module (CASFUM), which fuses trajectory confidence and detection confidence through an adaptive EMA mechanism, filters noise features of occluded area trajectories, and dynamically selects historical feature templates in conjunction with an AAAM correction module (AAAM), significantly improving the robustness of association under dense occlusion. However, its strong detection dependence and insufficient long-term modeling still restrict the tracking performance in complex dynamic scenes.

[0009] Patent CN117173221A innovatively introduces a realism grading strategy, distinguishing between high / low confidence detection targets through existence scores, and employs Recovery Overlap (RIoU) and an occluded pedestrian re-identification module to enhance the feature extraction and matching accuracy of occluded targets. This method significantly reduces false detection interference, but it does not fully exploit cross-frame temporal information.

[0010] In summary, current multi-target tracking methods still face core bottlenecks in complex scenarios, such as insufficient collaborative optimization of detection and association tasks and limited long-term temporal context modeling capabilities. These bottlenecks lead to problems such as limited detection accuracy, poor trajectory continuity, and frequent identity switching. A method that can unify the modeling of short-term motion prediction and long-term trajectory recovery is needed to improve tracking robustness in complex dynamic environments. Summary of the Invention

[0011] To address the aforementioned technical problems, the technical solution adopted in this invention is: a multi-target tracking method based on long- and short-term trajectory correlation. By collaboratively optimizing short-term motion prediction and long-term trajectory recovery, this method solves problems such as missed detection, false detection, identity switching, and trajectory breakage in occluded scenarios in existing technologies, thereby achieving robust tracking in complex scenarios.

[0012] This invention proposes a multi-target tracking method based on long- and short-term trajectory correlation, comprising the following core modules:

[0013] 1. Detection Module: The detection module uses the YOLOX detector to generate high-precision detection boxes in real time and outputs detection queries.

[0014] 2. Motion Prediction Module: The motion prediction module utilizes the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. The motion information includes the center coordinates. ,high ,width Confidence level And appearance information, etc.;

[0015] 3. Track Query Memory Module (TQMM): The TQMM module contains a spatiotemporal enhancement module, a query memory module, and an update module. The TQMM module enhances the trajectory information of the current frame through its internal spatiotemporal enhancement module, stores the trajectory information of the tracked target through its internal query memory module in conjunction with the update module, and continuously updates the information through the update module and a clearing mechanism to provide the latest trajectory information of lost targets for long-term association.

[0016] 4. Matching Module: The matching module performs IOU matching between the predicted bounding boxes generated by the motion prediction module and the detection bounding boxes generated by the detection module in each frame. Targets that successfully match are considered to have achieved short-term association.

[0017] 5. Historical Backtracking Module: The historical backtracking module calculates the correlation between unmatched detection targets in short-term association and the lost trajectories stored in the TQMM module, and performs cross-frame association on unmatched detections. Targets that are successfully associated are thus associated in the long term.

[0018] This invention proposes a multi-target tracking method based on long- and short-term trajectory correlation, which specifically includes the following steps:

[0019] 1) First, target detection is performed on the input video stream using the YOLOX detector, generating high-precision detection boxes in real time, and providing detection queries for long-term and short-term correlations;

[0020] 2) Using the MOTR-ConvNext network, the motion information of the tracked target is predicted through dynamically updated trajectory query, detection query and image features. At the same time, the Track Query Memory Module (TQMM) enhances, stores and updates trajectory information through its internal spatiotemporal enhancement module, query memory module and update module.

[0021] 3) The prediction bounding box and the detection bounding box of each frame are matched by IOU through the matching module to achieve short-term association;

[0022] 4) The historical backtracking module calculates the correlation between mismatched detections and memory trajectories in a lost state to achieve long-term association;

[0023] 5) Integrate the results of short-term and long-term correlations to obtain the complete tracking results for each frame.

[0024] Step 1) includes (taking frame t as an example):

[0025] 1-1) Given a video stream of t frames as input, pass it through a YOLOX detector to obtain the detection results for frame (t-1) and frame t. The YOLOX detector uses YOLOX weights provided from the official dataset. The detection result for each frame is determined by... Each bounding box represents a central coordinate system. ,high ,width and confidence level To indicate;

[0026] 1-2) By introducing shared queries, a set of proposed queries are generated from the bounding box. The proposed queries will replace the detection queries in MOTR-ConvNext to detect new targets. In fact, the generated proposed queries are the detection queries provided for long-short-term associations.

[0027] Step 2) includes (taking frame t as an example):

[0028] 2-1) The MOTR-ConvNext network obtains the trajectory query for frame t-1 through the MOTR-ConvNext network in frame t-2;

[0029] 2-2) The query obtained by concatenating the trajectory query and the detection query of frame t-1, along with the image features of frame t-1, is input into the decoder in MOTR-ConvNext;

[0030] 2-3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in frame t, and perform motion prediction on the tracked target. The hidden states are also fed back to the Query Interaction Module (QIM) in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame (frame t) based on the hidden state of the previous frame.

[0031] 2-4) The TQMM module first aggregates the trajectory information of historical frames and the current frame through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. Then, it stores the trajectory query information of all targets that have appeared through the internal query memory module combined with the internal update module. The trajectory information in TQMM is continuously updated through the update module and the internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association.

[0032] Steps 2-4) include:

[0033] 2-4-1) The spatiotemporal enhancement module, before aggregation The spatiotemporal enhancement module uses trajectory information from previous frames to enhance the trajectory information of the current frame. It consists of four layers, each comprising a self-attention layer, a matrix multiplication and softmax layer, a weighted summation layer, and a feedforward neural network layer. This invention converts timestamps into location embeddings to further refine the model's processing capabilities. A memory buffer is also used to store trajectory query information from multiple historical frames within the spatiotemporal enhancement module. This buffer employs a first-in-first-out (FIFO) data structure and has a maximum storage capacity of [missing information]. Frame duration, with a maximum of M object information items stored per frame;

[0034] 2-4-2) The current frame trajectory query with fused context information is compared with the memory trajectory query stored in the query memory module for similarity estimation, and then the information of the current trajectory query is stored through a storage mechanism; the historical trajectory queries of all tracked objects that have appeared are stored in the query memory module, and the memory module maintains the maximum length of the memory bank through a sliding window mechanism, with a window length of [missing information]. A frame can store information on a maximum of M objects;

[0035] 2-4-3) The trajectory queries that have not been updated for a long time are cleared through the clearing mechanism, thereby continuously updating the trajectory information in TQMM.

[0036] Step 2-4-1) includes:

[0037] 2-4-1-1) First, query the trajectory of the current frame (frame t). The information representation is enhanced through self-attention updates, where the dimension of each query in each frame is [dimension number missing]. The t-th frame has One trajectory query, the first The frames (historical frames) are respectively A trajectory query, the trajectory query for the current frame is... ,forward The trajectory queries for the frames are respectively Then, through a linear transformation, it is used as Q, while the historical trajectory query in the memory buffer, after a linear transformation, serves as both K and V:

[0038] (1)

[0039] Where Q, K, and V are the query, key, and value matrices, respectively. For historical trajectory queries, and For time encoding matrix, , and It is a linear transformation matrix. It is the number of trajectories, and also The maximum number of targets in a frame, for targets less than The frames are then filled with zero vectors. ;

[0040] 2-4-1-2) and Perform matrix multiplication, then normalize using a Softmax layer to obtain the similarity matrix. ,in It is the dimension of the key vector. This represents the dot product between the query and the key. This step measures the similarity between the query and all keys, and then divides by... To scale the dot product result, where It is a mask matrix used to ignore the effect of padding items;

[0041] 2-4-1-3) Weighting historical trajectory queries using a similarity matrix, the similarity matrix... Multiply To obtain a historical trajectory query that incorporates information from the current frame, i.e. ;

[0042] 2-4-1-4) Add the current trajectory query to the current historical trajectory query to generate a current trajectory query that incorporates historical frame information: ;

[0043] (2-4-1-5) Finally, a positional feedforward network (FFN) and layer normalization are used for further adjustments. The feedforward neural network (FFN) layer consists of two linear transformations and a ReLU activation function. First, the first linear transformation (FC layer) maps the weighted summation result to a high-dimensional space. Then, the ReLU activation function introduces non-linear characteristics to enhance the model's expressive power. Finally, the second linear transformation (FC layer) restores the high-dimensional features to the target dimension. Layer normalization (LN layer) performs a normalization operation on the features after FFN processing, eliminating data distribution differences between different layers to accelerate training convergence. After adjustment, the trajectory query with fused context information in frame t is finally obtained. , where FC represents a linear projection layer, LN represents layer normalization, and Re represents the ReLU activation function.

[0044] Step 3) includes:

[0045] 3-1) Calculate the Intersection over Union (IOU) value between the predicted bounding box and the detected bounding box. IOU quantifies the degree of overlap by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes. The formula is expressed as: ,in and These are the regions of the two bounding boxes. Indicates intersection, IOU represents the union of sets, and its value range is... A larger value indicates a higher degree of overlap.

[0046] 3-2) Set a threshold for judgment, set the IOU threshold. The system determines whether the predicted bounding box matches the detected bounding box. If the Interchange of Union (IOU) is greater than the threshold, the two are considered to be successfully matched, and the target achieves short-term association. Otherwise, the two are not successfully matched, and the detection needs to be associated in the long term.

[0047] Step 4) includes (taking frame t as an example):

[0048] 4-1) Assume that there are d mismatched detections after short-term association in frame t, and query the corresponding detections. Querying the memory trajectory that is in a lost state stored in the TQMM module The correlation is calculated (assuming the TQMM stores t memory trajectories in a lost state), and the correlation matrix is ​​obtained through the calculation module. ;

[0049] 4-2) Compare the elements of each column or row of the correlation matrix to find the element with the highest value in each column or row. Each element in the array represents the probability of association between an unmatched detected target and another memory trajectory in a lost state. That is, two queries with high correlation correspond to the same target. A successfully matched target is associated with the target for a long time. If no match is found, that is, the correlation between the detection query of the unmatched detected target and the trajectory queries of all targets in a lost state stored in the memory module is low, then the detected target is considered a new target and used for target prediction in the next frame to generate a trajectory query for the new target.

[0050] Step 4-1) includes:

[0051] 4-1-1) First, the two queries are processed through an independent self-attention mechanism to capture the dependencies within the sequence, generating enhanced feature representations. The self-attention formula is as follows: Where Q, K, and V are the query, key, and value matrices, respectively. It is the dimension of the key, used to scale attention weights;

[0052] 4-1-2) Padding and masking are used to ensure that the two input tensors are of the same size, while providing information to ignore invalid padding data, thus obtaining two trajectory queries of the same size through padding. , , The number of trajectories after filling the zero vector is also the maximum number of targets in the two trajectory queries (assuming that the memory trajectory has the most targets), and D is the dimension of each query;

[0053] 4-1-3) Querying lost memory trajectory The query (Q) is obtained after a linear transformation, i.e. , will detect query The bond (K) is obtained after a linear transformation, i.e. ,in and It is a linear transformation matrix;

[0054] 4-1-4) Then, borrowing the method of cross-attention, we can calculate the relevance, allowing each element in one sequence to pay attention to all elements in another sequence and obtain the degree of this attention. First, we... and Perform matrix multiplication, and then normalize using the Softmax function to obtain the correlation matrix. ,in It is a mask matrix, still used to ignore the influence of padding terms, through the correlation matrix. You can get a query for detecting unmatched targets. Each trajectory in the memory trajectory query is in a lost state. The correlation of each trajectory in the data.

[0055] The beneficial effects of adopting the above technical solution are as follows:

[0056] 1. By combining the YOLOX detector and the MOTR-ConvNext model, dynamic trajectory queries and detection queries are used to jointly model the motion and appearance features between adjacent frames to optimize short-term motion prediction. Specifically, the ConvNeXt backbone network enhances the contextual awareness of occluded targets by expanding the receptive field and incorporating hierarchical feature fusion.

[0057] 2. Design a trajectory query and memory module. This module stores and updates historical trajectory information based on spatiotemporal augmentation mechanism and exponential moving average (EMA) strategy, and achieves robust matching of long-term lost targets through multi-frame context awareness.

[0058] 3. A historical backtracking module is introduced to perform global cross-frame association of unmatched detections, enabling the recovery and identity preservation of long-lost targets. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 The overall framework diagram of a multi-target tracking method based on long-short trajectory correlation provided by the present invention;

[0061] Figure 2 A flowchart illustrating a multi-target tracking method based on long- and short-term trajectory correlation provided by the present invention;

[0062] Figure 3 This is a structural diagram of the MOTR-ConvNext network provided by the present invention.

[0063] Figure 4 This is a structural diagram of the TQMM module provided by the present invention.

[0064] Figure 5 This is a visualization of the results of a multi-target tracking method based on long-short trajectory correlation provided in this invention application on the MOT17 dataset.

[0065] Figure 6This is a visualization result of a multi-target tracking method based on long-short trajectory correlation provided in this invention application on the DanceTrack dataset. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0067] Figure 1 The present invention provides an overall framework diagram of a multi-target tracking method based on long-short trajectory correlation. This method achieves this goal by first performing target detection on each video frame and then performing two steps: short-term correlation and long-term correlation.

[0068] 1. Target detection: This step is completed by the detection module, which uses the YOLOX detector to generate high-precision detection boxes in real time and provides detection queries for long-term and short-term correlations.

[0069] 2. Short-term association: This step includes a motion prediction module, a track query memory module (TQMM), and a matching module. The motion prediction module utilizes the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated track queries, detection queries, and image features. The MOTR-ConvNext network iteratively predicts the target trajectory sequence, and the motion information includes the center coordinates. ,high ,width Confidence level In addition to appearance information, the TQMM module enhances the trajectory information of the current frame through a spatiotemporal enhancement module, stores the trajectory information of the tracked target through a query memory module combined with an update module, and continuously updates the information through the update module and an internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association. The matching module performs IOU matching between the predicted bounding boxes generated by the motion prediction module and the detection bounding boxes generated by the detection module in each frame. Targets that successfully match achieve short-term association.

[0070] 3. Long-term correlation: Long-term correlation is accomplished by the historical backtracking module. It calculates the correlation between the unsuccessfully matched detections in the short-term correlation and the lost trajectories stored in the TQMM module, and generates a correlation matrix. By using correlation scores, unmatched detections are associated across frames, and the goal of successful association is to achieve long-term association.

[0071] Figure 2This is a flowchart illustrating a multi-target tracking method based on long-short-term trajectory correlation provided by the present invention. Combining the above-mentioned multi-target tracking method framework, a multi-target tracking method based on long-short-term trajectory correlation can be implemented. The specific steps are as follows (taking frame t as an example):

[0072] Step S201 first uses the YOLOX detector to perform target detection on the input video stream, generating high-precision detection bounding boxes in real time, and providing detection queries for long-term and short-term correlations. Specific steps include:

[0073] (1) Pass the given input t frame video stream through the YOLOX detector to obtain the detection result of the (t-1)th frame. and the detection results of frame t The YOLOX detector uses YOLOX weights provided from the official dataset, and the detection results for each frame are determined by... Each bounding box represents a central coordinate system. ,high ,width and confidence level To express.

[0074] (2) By introducing a shared query, a set of proposed queries are generated from the bounding box. The proposed queries will replace the detection queries in MOTR-ConvNext. To detect newly generated targets, L represents the number of newly generated targets, and D represents the dimension of each trajectory. Figure 1 The detection query output by the target detection process is the detection query after being replaced by the proposed query.

[0075] Step S202 utilizes the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. Simultaneously, the TQMM module stores and updates the trajectory information through its internal query memory module and update module. Specific steps include:

[0076] (1) The MOTR-ConvNext network obtains the trajectory query of the (t-1)th frame through the MOTR-ConvNext network of the (t-2)th frame. ( (The number of trajectories in frame t-1).

[0077] (2) Query the trajectory of the (t-1)th frame. and the detection query of frame t-1 ( (Number of new targets in frame t-1), the concatenated query, and the image features of frame t-1. They are input together into the decoder in MOTR-ConvNext.

[0078] (3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in frame t, perform motion prediction on the tracked target, and the hidden states are also fed back to the Query Interaction Module (QIM) in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame (frame t) based on the hidden state of the previous frame.

[0079] (4) The TQMM module first aggregates the trajectory information of historical frames and current frames through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. Then, it stores the trajectory query information of all targets that have appeared through the internal query memory module combined with the internal update module. The trajectory information in TQMM is continuously updated through the update module and the internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association.

[0080] Step S203 uses the matching module to perform IOU matching between the predicted bounding box and the detected bounding box in frame t, achieving short-term association. Specific steps include:

[0081] (1) Calculate the Intersection over Union (IOU) value between the predicted bounding box and the detected bounding box. IOU matching is a core algorithm in object detection and multi-object tracking, used to measure the degree of spatial overlap between two bounding boxes to determine whether they belong to the same object. IOU quantifies the degree of overlap by calculating the ratio of the intersection area to the union area of ​​two bounding boxes. The formula is expressed as: ,in and These are the regions of the two bounding boxes. Indicates intersection, Represents the union of sets. The range of values ​​for IOU is... A higher value indicates a higher degree of overlap.

[0082] (2) Set a threshold for judgment. Set the IOU threshold. The system determines whether the predicted bounding box matches the detected bounding box. If the Intersection over Union (IOU) is greater than or equal to the threshold, the two are considered to have matched successfully, and the target has achieved a short-term association. Otherwise, the two have not matched successfully, and the detection needs to be associated in the long term.

[0083] Step S204 uses the historical backtracking module to establish a long-term association between the mismatched detections and the trajectories in a lost state. Specific steps include:

[0084] (1) Assume that there are d mismatched detections after the short-term association of the t-th frame, and query the corresponding detections. Query the memory trajectory that is in a lost state stored in the TQMM module. The correlation is calculated (assuming the TQMM stores t memory trajectories in a lost state), where D is the dimension of each trajectory, and the correlation matrix is ​​obtained through the calculation module. The correlation calculation process is as follows:

[0085] ① First, the two queries are processed through an independent self-attention mechanism to capture the dependencies within the sequence, generating enhanced feature representations. The self-attention formula is as follows: , where Q, K and V are the query, key and value matrices, respectively. It is the dimension of the key, used to scale attention weights.

[0086] ② Use padding and masking to ensure that the two input tensors are the same size, while providing information to ignore invalid padding data, and obtain two trajectory queries of the same size through padding. , , The number of trajectories after filling the zero vector is also the maximum number of targets in the two trajectory queries (assuming the memory trajectory has the most targets), and D is the dimension of each query.

[0087] ③ Search for lost memory trajectory The query (Q) is obtained after a linear transformation, i.e. , will detect query The bond (K) is obtained after a linear transformation, i.e. ,in and It is a linear transformation matrix.

[0088] ④ Then, the cross-attention method is used to calculate relevance, allowing each element in one sequence to pay attention to all elements in another sequence and obtain the degree of this attention. First, for... and Perform matrix multiplication, and then normalize using the Softmax function to obtain the correlation matrix. ,in It's a mask matrix, still used to ignore the effect of padding terms. This is achieved through the correlation matrix. We can obtain the detection query for unmatched targets. Each trajectory in the memory trajectory query is in a lost state. The correlation of each trajectory in the data.

[0089] (2) Compare the elements of each column or row of the correlation matrix to find the element with the highest value in each column or row. The correlation matrix... Each element in the memory module represents the probability of association between an unmatched detected target and another memory trajectory in a lost state. This means that two queries with high correlation correspond to the same target, and a successful match establishes a long-term association. If no match is found, meaning the unmatched detection has low correlation with the trajectories of all targets in a lost state stored in the memory module, then the detected target is considered a new target and used for target prediction in the next frame to generate a new target trajectory query.

[0090] Step S205 combines the results of short-term and long-term correlations to output the complete tracking results for frame t.

[0091] Figure 3 The diagram shows the structure of the MOTR-ConvNext network provided by this invention. The MOTR-ConvNext network iteratively processes each frame of video, predicting the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. The specific steps are as follows:

[0092] Step 1: The video sequence is first fed into the ConvNext backbone network (the present invention actually uses the ConvNext-Base network) for feature extraction. This network is based on a pure convolutional architecture, combining the global modeling capability of Transformer with the local perception characteristics of CNN. It extracts multi-scale semantic features of the image layer by layer by stacking multiple layers of convolutional blocks in four stages (including depthwise separable convolution, layer normalization and GELU activation function). The ConvNeXt-Base network used is a medium-sized version of this series, with approximately 88M parameters. The number of channels in the four stages is set to 128, 256, 512 and 1024 respectively, and the ratio of the number of modules in each stage is 1:1:3:1. This design enhances the ability to extract abstract semantic features while preserving local details, and is especially suitable for feature encoding of high-resolution frames in video sequence processing. The extracted features are then fed into the Deformable DETR encoder (Encoder, abbreviated as Enc). The core improvement of Enc lies in the introduction of a deformable attention mechanism, which reduces computational complexity by sparsely sampling feature points at key locations while retaining the ability to perceive multi-scale targets. Internally, the encoder aggregates global contextual information through multiple layers of self-attention and cross-attention modules, and combines sine and cosine positional encoding to model spatial relationships, ultimately generating image features with semantic and spatial information. , here Indicates the first The characteristics of the frames are obtained by the video sequence passing through the YOLOX detector to obtain the detection query. To detect emerging targets, the YOLOX detector features a lightweight, anchor-free design. It extracts multi-level features through the CSPDarknet backbone network, then fuses semantic information at different scales via a Feature Pyramid Network (FPN). Decoupled heads then handle classification and regression tasks separately. The classification branch predicts target class probabilities through multi-layer convolutions, while the regression branch directly outputs bounding box coordinates. The SimOTA label allocation strategy optimizes positive and negative sample matching, achieving efficient and accurate real-time target detection.

[0093] Step 2: The Query Interaction Module (QIM) queries based on the initial trajectory of the previous frame. Generate trajectory query ( (Generated from initialization), where the QIM module is a component designed based on the Transformer architecture. The QIM structure includes an object entry and exit mechanism and a Temporal Aggregation Network (TAN), which dynamically maintains the trajectory query set of targets in the video sequence and enhances the modeling of long-term sequences. QIM dynamically filters high-scoring trajectory queries from the initial trajectory queries (hidden states) generated by the Transformer decoder through the object entry and exit mechanism. Simultaneously, the Temporal Aggregation Network uses a multi-head self-attention mechanism to interact with the hidden state of the current frame and the historical trajectory information of the previous frame, aggregating long-term temporal features to enhance the modeling ability of target motion, thereby generating the trajectory query for the next frame. Then the trajectory query... and the detection query corresponding to each frame Cascaded together, the cascaded queries and features Together, these are fed into the decoder (Decoder). The Dec is a modular structure based on the Transformer architecture, employing a multi-layered Transformer structure, including a self-attention mechanism and an encoder-decoder attention mechanism (Cross-Attention). The former captures the temporal correlation between the query and features by calculating the similarity between the query, key, and value; the latter uses the feature map extracted by the encoder as contextual information and dynamically locates the target's region in the feature map through attention weights. The decoder also includes a feedforward neural network (FFN) to perform nonlinear transformations and feature fusion on the attention output information to enhance the model's expressive power, finally generating the initial trajectory query for the current frame. Initial trajectory query In the hidden state, its function is to predict the bounding box of the tracked object in the next frame and perform motion prediction on the tracked target.

[0094] Step 3: Then through Generate the prediction result for the current frame. Prediction results Including center coordinates ,high ,width and confidence level Information such as the initial trajectory query will be provided. Feedback is sent to QIM, and then generated through QIM updates. Send it to the next frame. Among them, for the first... The first frame (frame) is initialized to empty for trajectory query, containing only features. and detection query The input is entered into Dec for interaction and to generate the first... Initial trajectory query of the frame Then, a prediction for the current frame is generated. And the next frame (the first) Trajectory query of frames .

[0095] Figure 4 The TQMM module structure diagram provided by this invention is shown below. The TQMM module first enhances the current trajectory information through an internal spatiotemporal enhancement module, then stores trajectory query information for all previously encountered targets through an internal query memory module and update module, and continuously updates the trajectory information in the TQMM through an update module and an internal clearing mechanism. The specific steps are as follows:

[0096] Step 1: The spatiotemporal enhancement module passes the aggregation process before... The trajectory information of the current frame is enhanced by comparing the trajectory information of the previous frame with that of the current frame. The spatiotemporal enhancement module consists of four layers, each composed of a self-attention layer, a matrix multiplication and softmax layer, a weighted summation layer, and a feedforward neural network layer. This invention converts timestamps into location embeddings to further refine the model's processing capabilities, and simultaneously sets up a memory buffer to store trajectory query information from multiple historical frames in the spatiotemporal enhancement module. The buffer can store a maximum of [number missing] frames. The frame duration is specified, and each frame stores a maximum of M object information. This buffer is implemented using a first-in, first-out (FIFO) data structure; as new frame information is added, the oldest data in the queue is discarded.

[0097] Step 1.1: First, query the trajectory of the current frame (frame t). The information representation is enhanced through self-attention layer updates. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence. This mechanism allows the model to dynamically adjust the level of attention given to each element when processing sequence data, thereby capturing complex dependencies within the sequence and enhancing the information representation. In this step, the dimension of each query in each frame is... The t-th frame has One trajectory query, the first The frames (historical frames) are respectively A trajectory query, the trajectory query for the current frame is... ,forward The trajectory queries for frames (historical frames) are as follows: Then, through a linear transformation, it is used as Q, while the historical trajectory query in the memory buffer, after a linear transformation, serves as both K and V:

[0098] (2)

[0099] Where Q, K, and V are the query, key, and value matrices, respectively. For historical trajectory queries, and For time encoding matrix, , and It is a linear transformation matrix. It is the number of trajectories, and also The maximum number of targets in a frame. For frames with fewer than [number of targets]... The frames are then filled with zero vectors. .

[0100] Step 1.2: For and Perform matrix multiplication, then normalize using a Softmax layer to obtain the similarity matrix. ,in It is the dimension of the key vector. This represents the dot product between the query and the key. This step measures the similarity between the query and all keys, and then divides by... To scale the dot product result, where It is a mask matrix used to ignore the effect of padding.

[0101] Step 1.3: Weight the historical trajectory query using a similarity matrix. Multiply To obtain a historical trajectory query that incorporates information from the current frame, i.e. .

[0102] Step 1.4: Add the current trajectory query to the current historical trajectory query to generate a current trajectory query that incorporates historical frame information: .

[0103] Step 1.5: Finally, further adjustments are made using a positional feedforward network (FFN) and layer normalization. The feedforward neural network (FFN) layer consists of two linear transformations and a ReLU activation function. First, the first linear transformation (FC layer) maps the weighted summation result to a high-dimensional space. Then, the ReLU activation function introduces non-linear characteristics to enhance the model's expressive power. Finally, the second linear transformation (FC layer) restores the high-dimensional features to the target dimension. Layer normalization (LN layer) performs a normalization operation on the features after FFN processing, eliminating data distribution differences between different layers to accelerate training convergence. After adjustment, the trajectory query with fused context information in frame t is finally obtained. , where FC represents a linear projection layer, LN represents layer normalization, and Re represents the ReLU activation function.

[0104] Step 2: The current frame trajectory query with fused context information is compared with the memory trajectory queries stored in the query memory module for similarity estimation. Then, the information of the current trajectory query is stored through a storage mechanism. Historical trajectory queries of all tracked objects that have appeared are stored in the query memory module. The memory module maintains the maximum length of the memory bank through a sliding window mechanism, with a window length of [missing information]. A frame can store information on a maximum of M objects.

[0105] Step 2.1: First, use cosine similarity to match each trajectory in the current trajectory query with each trajectory in the memory trajectory query to obtain a similarity score. Matrix operations are used to calculate the similarity between all trajectory query and memory trajectory query entries at once, thus improving calculation speed and ultimately obtaining a similarity matrix. The cosine similarity formula is: ,in Indicates current trajectory query The i-th trajectory and memory trajectory query The j-th trajectory The cosine similarity is given by M and N, where M and N are the number of trajectories and D is the vector dimension.

[0106] Step 2.2: Query memory module update by setting a similarity threshold. The system determines whether the target in the current frame is a new target, and then updates the memory trajectory query in the query memory module according to the update mechanism. If the target similarity in the current frame is greater than or equal to the similarity threshold S, then the target is an old target (a target that has already been tracked). The trajectory information of this target is then smoothly overwritten with the corresponding information in the memory trajectory query, and an unupdated counter (counting the number of frames where the target information has not been updated) is initialized. Since direct overwriting may cause abrupt changes in feature information, an exponential moving average (EMA) is used to smooth the information update. ,in The exponential moving average over time t. It is the actual data value of time t. It is the exponential moving average over time t-1. It is a smoothing factor hyperparameter, usually If the similarity of a target in the current frame is less than the similarity threshold S, then the target is a new target (a newly appearing target), and the trajectory information corresponding to the target is stored in the query memory module, while the unupdated counter is initialized.

[0107] Step 3: Clear long-unupdated trajectory queries through a clearing mechanism, thereby continuously updating the trajectory information in TQMM.

[0108] Regarding the cleanup mechanism, the trajectory query in the memory module is divided into three states: alive, lost, and inactive. A frame threshold is set through the cleanup mechanism. This is used to determine the status of the memory trajectory query in the memory module. The unupdated counter is updated frame by frame. If the unupdated counter value of a target in the memory trajectory query is 0, then the target is alive, denoted as . If the value of the unupdated counter for a target in the memory trajectory query is greater than 1 and less than or equal to 1... If the target is lost, it is recorded as... If the value of the unupdated counter for a certain target in the memory trajectory query is greater than ; If the target is deactivated, the memory module will delete the target's trajectory information.

[0109] To achieve effective training for multi-target tracking, we designed a comprehensive loss function that considers both short-term and long-term correlation factors.

[0110] For short-term association, the loss function for frame t is expressed as: ,in, It is the classification loss in short-term association, calculated using focal loss; It is a short-term association loss; It is the generalized crossover ratio loss in short-term correlation. , and These are the corresponding weighting coefficients. For long-term correlations, this invention considers that the correlation matrix obtained from the historical backtracking module is... The loss function for long-term association is defined as follows: (Assume there are d mismatched detections after short-term association in frame t, and TQMM stores t memory trajectories in a lost state). Here The dissimilarity between the detection query that does not match the target and the lost memory trajectory query is represented by summing all dissimilarity values ​​to obtain the long-term association loss.

[0111] Finally, the total loss function of this method is a weighted sum of short-term association loss and long-term association loss. ,in and These are the weights of the short-term correlation loss and the long-term correlation loss, used to balance the importance of the two types of correlation during training. Through the design of the loss function, this method can simultaneously optimize the performance of short-term and long-term correlations during training, thereby improving the overall performance of multi-object tracking.

[0112] To fully validate the performance of this method, experiments were conducted on two mainstream multi-object tracking datasets, MOT17 and DanceTrack, comparing it with previous state-of-the-art methods. Table 1 shows the performance comparison of this method with other state-of-the-art methods on the DanceTrack test set (↑ indicates that higher metrics are better). It should be noted that the experimental data for other state-of-the-art methods are all cited from their respective original papers.

[0113] Table 1 Comparison with existing methods on the DanceTrack dataset

[0114] CNN-based FairMOT 39.7 23.8 66.7 82.2 40.8 ByteTrack 47.7 32.1 71.0 89.6 53.9 OC-SORT 55.1 38.3 80.3 92.0 54.6 Transformer-based TransTrack 45.5 27.5 75.9 88.4 45.2 MOTR 54.2 40.2 73.5 79.7 51.5 MOTRv2 69.9 59.0 83.0 91.9 71.7 MOTRv3 70.4 59.3 83.8 92.9 72.3 This method 70.3 60.1 83.4 91.4 73.8

[0115] Table 1 shows that the IDF1 of our proposed method is 1.5 higher than MOTRv3 (72.3) and 2.1 higher than MOTRv2 (71.7), validating the effectiveness of the long-term association module. MOTRv3 is a Transformer-based end-to-end method proposed in the paper "MOTRv3: Release-FetchSupervision for End-to-End Multi-Object Tracking," while MOTRv2 is a Transformer-based end-to-end method proposed in the paper "MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors." We attribute this to the TQMM module's dynamic maintenance of the memory bank through EMA smooth updates and sliding window cleanup, restoring the target's identity through the history backtracking module upon target reappearance. Our method's AssA (60.1) is higher than MOTRv3's (59.3), indicating that its trajectory ID consistency over time is more robust. We attribute this to the hierarchical feature enhancement of the ConvNeXt backbone network for modeling complex motions, combined with the spatiotemporal enhancement module of TQMM, and the optimization of trajectory continuity through multi-frame attention fusion. Our method achieves a HOTA 0.4 higher than MOTRv2 and slightly lower than MOTRv3 by 0.1, but its association accuracy (60.1) is significantly better than both MOTRv3 (59.3) and MOTRv2 (59.0). Regarding the comprehensive detection (DetA) and association (AssA) performance of HOTA, our method's long-term association module reduces ID switching through historical trajectory recovery (resulting in an improved IDF1), but the introduced trajectory noise causes a slight decrease in DetA, which in turn leads to a slight decrease in HOTA. Our method's MOTA (91.2) is slightly lower than MOTRv2 (91.9). We speculate that the MOTA metric focuses more on detection and short-term tracking, neglecting trajectory continuity, while our method has a significant advantage in long-term association, resulting in a slightly lower MOTA.

[0116] Table 2 shows the performance comparison of our method with other state-of-the-art (SOTA) methods on the MOT17 test set (↑ / ↓ indicate higher / lower metrics are better). It should be noted that the experimental data for other SOTA methods are all cited from their respective original papers.

[0117] Table 2 Comparison with existing methods on the MOT17 dataset

[0118] CNN-based FairMOT 59.3 58.0 60.9 73.7 72.3 3303 ByteTrack 63.1 62.0 64.5 80.3 77.3 2196 OC-SORT 63.2 63.2 / 78.0 77.5 / Transformer-based TransTrack 54.1 47.9 61.6 74.5 63.9 3663 MOTR 57.8 55.7 60.3 73.4 68.6 2439 MOTRv3 60.2 58.7 62.1 75.9 72.4 2403 MOTRV2 62.0 60.6 63.8 78.6 75.0 / This method 62.9 61.8 64.4 77.8 77.2 1807

[0119] Table 2 shows that our proposed method outperforms MOTRv3, with a 2.7 improvement in HOTA and a 4.8 improvement in IDF1. Our method's IDS is 33% lower than MOTRv3, indicating that the obtained trajectories are continuous and robust. Compared to MOTRv2, our method also performs better. However, similar to the results on the DanceTrack test set, our method's MOTA is still slightly lower than MOTRv2. We still speculate that our method's significant advantage in long-term correlation leads to its slightly lower MOTA. Furthermore, we can see that CNN-based methods (such as ByteTrack and OC-SORT) perform well on the MOT17 dataset, although they perform worse than our method on DanceTrack. We infer that this is because the target motion trajectories in MOT17 are simple, thus, by combining powerful target detectors such as YOLOX and well-designed post-processing rules, targets in MOT17 can be tracked effectively.

[0120] To demonstrate the tracking effectiveness of the method proposed in this application, visualizations of the algorithm proposed in this application and the classic MOTRv2 algorithm on the MOT17 dataset and the DanceTrack dataset are presented.

[0121] On the MOT17 dataset, such as Figure 5 As shown in (a), pedestrian number 41 was occluded in frame 234 of MOTRv2. In frame 267, pedestrian number 41 reappeared, but because MOTRv2 relies on short-term matching and lacks long-term association, the ID of the original pedestrian number 41 changed from 41 to 72. Simultaneously, the original pedestrian number 72 was occluded. When the original pedestrian number 72 reappeared in frame 365, its ID changed from 72 to 126. And from... Figure 5 (b) It can be clearly seen that the IDs of all pedestrians in each frame of this method do not change.

[0122] On the DanceTrack dataset, such as Figure 6 As shown in (a), dancer #4 is occluded at frame 60 in MOTRv2. Dancer #4 reappears at frame 60, and its ID changes from 4 to 6. Occlusion occurs again at frame 115. When all dancers reappear at frame 174, the ID of dancer #1 changes from 1 to 7, the ID of dancer #3 changes from 2 to 8, and the ID of dancer #5 changes from 5 to 9. And from... Figure 6 (b) It can be clearly seen that the IDs of all dancers remain unchanged in each frame of this method.

[0123] Through the above visualization and analysis, we can see that this method significantly reduces the ID switch phenomenon in complex scenarios (such as DanceTrack and MOT17) by associating long and short trajectories, indicating that the model has significant advantages in complex motion modeling and identity preservation.

[0124] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A multi-target tracking method based on long- and short-term trajectory correlation, characterized in that, include: 1) First, target detection is performed on the input video stream using the YOLOX detector, generating high-precision detection boxes in real time, and providing detection queries for long-term and short-term correlations; 2) The MOTR-ConvNext network is used, which includes a ConvNext backbone network, a Deformable DETR encoder, a query interaction module, and a decoder. The ConvNext backbone network and the Deformable DETR encoder are used to extract image features of the input video frame. The query interaction module is used to dynamically update the trajectory query based on the hidden state of the previous frame. The detection query is provided by the proposal query generated by the YOLOX detector. The decoder is used to interact with the stitched trajectory query, detection query and image features to generate a hidden state to predict the motion information of the tracked target. At the same time, the trajectory query memory module TQMM module first aggregates the trajectory information of the historical frame and the current frame through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. Then, it stores the trajectory query information of all targets that have appeared through the internal query memory module combined with the internal update module. The trajectory information in TQMM is continuously updated through the update module and the internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association. 3) The prediction bounding box and the detection bounding box of each frame are matched by IOU through the matching module to achieve short-term association; 4) The historical backtracking module calculates the correlation between mismatched detections and lost memory trajectories to achieve long-term association, including: 4-1) Assume that there are d mismatched detections after short-term association in frame t, and query the corresponding detections. Querying the memory trajectory that is in a lost state stored in the TQMM module Correlation calculations are performed, assuming that the TQMM stores t memory trajectories in a lost state, and the correlation matrix is ​​obtained through the calculation module. ; 4-2) Compare the elements of each column or row of the correlation matrix to find the element with the highest value in each column or row. Each element in the array represents the probability of association between an unmatched detected target and another memory trajectory in a lost state. That is, two queries with high correlation correspond to the same target. A successfully matched target is associated with a long-term target. If a match is still not successful, that is, the correlation between the detection query of the unmatched detected target and the trajectory queries of all targets in a lost state stored in the memory module is low, then the detected target is considered a new target and used for target prediction in the next frame to generate a trajectory query for the new target. 5) Integrate the results of short-term and long-term correlations to obtain the complete tracking results for each frame.

2. The multi-target tracking method based on long- and short-term trajectory correlation according to claim 1, characterized in that, Step 1) includes: 1-1) Given a video stream of t frames as input, pass it through a YOLOX detector to obtain the detection results for frame (t-1) and frame t. The YOLOX detector uses YOLOX weights provided from the official dataset. The detection result for each frame is determined by... Each bounding box represents a central coordinate system. ,high ,width and confidence level To indicate; 1-2) By introducing shared queries, a set of proposed queries are generated from the bounding box. The proposed queries will replace the detection queries in MOTR-ConvNext to detect new targets. In fact, the generated proposed queries are the detection queries provided by long-short association.

3. The multi-target tracking method based on long- and short-term trajectory correlation according to claim 1, characterized in that, Step 2) further includes: 2-1) The MOTR-ConvNext network obtains the trajectory query for frame t-1 through the MOTR-ConvNext network in frame t-2; 2-2) The query obtained by concatenating the trajectory query and the detection query of frame t-1, along with the image features of frame t-1, is input into the decoder in MOTR-ConvNext; 2-3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in frame t, and perform motion prediction on the tracked target. The hidden states are also fed back to the query interaction module QIM in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame t based on the hidden state of the previous frame.

4. The multi-target tracking method based on long-short-term trajectory correlation according to claim 3, characterized in that, The TQMM module first enhances the trajectory information of the current frame by aggregating trajectory information from historical frames and the current frame through a spatiotemporal enhancement module. Then, it stores trajectory query information for all targets that have appeared through an internal query memory module combined with an internal update module. Finally, it continuously updates the trajectory information in TQMM through the update module and an internal clearing mechanism, providing the latest trajectory information of lost targets for long-term association, including: 2-4-1) The spatiotemporal enhancement module, before aggregation The spatiotemporal enhancement module uses trajectory information from previous frames to enhance the trajectory information of the current frame. It consists of four layers, each comprising a self-attention layer, a matrix multiplication and softmax layer, a weighted summation layer, and a feedforward neural network layer. Timestamps are converted into location embeddings, and a memory buffer is set up to store trajectory query information from multiple historical frames in the spatiotemporal enhancement module. This memory buffer is implemented using a first-in-first-out (FIFO) data structure and has a maximum storage capacity of [missing information]. Frame duration, maximum number of frames saved per frame Information about each object; 2-4-2) The current frame trajectory query with fused context information is compared with the memory trajectory query stored in the query memory module for similarity estimation, and then the information of the current trajectory query is stored through a storage mechanism; the historical trajectory queries of all tracked objects that have appeared are stored in the query memory module, and the memory module maintains the maximum length of the memory bank through a sliding window mechanism, with a window length of [missing information]. Frames, at most Information about each object; 2-4-3) The trajectory queries that have not been updated for a long time are cleared through the clearing mechanism, thereby continuously updating the trajectory information in TQMM.

5. The multi-target tracking method based on long-short-term trajectory correlation according to claim 4, characterized in that, Step 2-4-1) includes: 2-4-1-1) First, query the trajectory of the current frame in frame t. The information representation is enhanced through self-attention updates, where the dimension of each query in each frame is [dimension number missing]. The t-th frame has One trajectory query, the first Each historical frame has A trajectory query, the trajectory query for the current frame is... ,forward The trajectory queries for the frames are respectively Then, through a linear transformation, it is used as Q, while the historical trajectory query in the memory buffer, after a linear transformation, serves as both K and V: （1） Where Q, K, and V are the query, key, and value matrices, respectively. For historical trajectory query matrix, It is the number of trajectories, and also The maximum number of targets in a frame, for targets less than The frames are then filled with zero vectors. ; 2-4-1-2) and Perform matrix multiplication, then normalize using a Softmax layer to obtain the similarity matrix. ,in It is the dimension of the key vector. This represents the dot product between the query and the key. This step measures the similarity between the query and all keys, and then divides by... To scale the dot product result, where It is a mask matrix used to ignore the effect of padding items; 2-4-1-3) Weighting historical trajectory queries using a similarity matrix, the similarity matrix... Multiply To obtain a historical trajectory query that incorporates information from the current frame, i.e. ; 2-4-1-4) Add the current trajectory query to the current historical trajectory query to generate a current trajectory query that incorporates historical frame information: ; (2-4-1-5) Finally, the position-based feedforward network (FFN) and layer normalization are used for further adjustments. The FFN layer of the feedforward neural network consists of two linear transformations and a ReLU activation function. First, the first linear transformation (FC) layer maps the weighted summation result to a high-dimensional space. Then, the ReLU activation function introduces non-linear characteristics to enhance the model's expressive power. Finally, the second linear transformation (FC) layer restores the high-dimensional features to the target dimension. The LN layer, after FFN processing, performs a standardization operation on the features to eliminate data distribution differences between different layers, thereby accelerating training convergence. After adjustment, the trajectory query with fused context information in frame t is finally obtained. , where FC represents a linear projection layer, LN represents layer normalization, and Re represents the ReLU activation function.

6. The multi-target tracking method based on long-short-term trajectory correlation according to claim 1, characterized in that, Step 3) includes: 3-1) Calculate the Intersection over Union (IOU) value between the predicted bounding box and the detected bounding box. IOU quantifies the degree of overlap by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes. The formula is expressed as: ,in and These are the regions of the two bounding boxes. Indicates intersection, IOU represents the union of sets, and its value range is... A larger value indicates a higher degree of overlap. 3-2) Set a threshold for judgment, set the IOU threshold. The system determines whether the predicted bounding box matches the detected bounding box. If the Interchange of Value (IOU) is greater than the threshold, the two are considered to be successfully matched, and the target achieves short-term association. Otherwise, the two are not successfully matched, and the detection needs to be associated in the long term.

7. The multi-target tracking method based on long- and short-term trajectory correlation according to claim 1, characterized in that, Step 4-1) includes: 4-1-1) First, the two queries are processed through an independent self-attention mechanism to capture the dependencies within the sequence, generating enhanced feature representations. The self-attention formula is as follows: Where Q, K, and V are the query, key, and value matrices, respectively. It is the dimension of the key, used to scale attention weights; 4-1-2) Padding and masking are used to ensure that the two input tensors are of the same size, while providing information to ignore invalid padding data, thus obtaining two trajectory queries of the same size through padding. , , The number of trajectories after filling the zero vector is also the maximum number of targets in the two trajectory queries. Assuming that the memory trajectory has the most targets, D is the dimension of each query. 4-1-3) Querying lost memory trajectory The query Q is obtained after a linear transformation, i.e. , will detect query The bond K is obtained after a linear transformation, i.e. ,in and It is a linear transformation matrix; 4-1-4) Then, borrowing the method of cross-attention, we can calculate the relevance, allowing each element in one sequence to pay attention to all elements in another sequence and obtain the degree of this attention. First, we... and Perform matrix multiplication, and then normalize using the Softmax function to obtain the correlation matrix. ,in It is a mask matrix, still used to ignore the influence of padding terms; through the correlation matrix Resulting in a query for detecting unmatched targets Each trajectory in the memory trajectory query is in a lost state. The correlation of each trajectory in the data.

Images