Multi-target tracking method based on long and short term trajectory association

Abstract

The invention belongs to the technical field of target tracking, and discloses a multi-target tracking method based on long and short term trajectory association. The method aims at solving the problems of precision limitation, track breakage, frequent identity switching and the like caused by insufficient collaborative optimization of detection and tracking tasks in a complex scene and limited long-term track modeling capability in an existing method. The invention provides a multi-target tracking method based on long and short term trajectory association. The method comprises the following steps: generating a target detection frame and detection query through a detection module based on YOLOX; the MOTR-ConvNext network dynamically updates the track query and predicts the target motion information; a track query memory module is used for storing and updating a historical track; the detection frame and the prediction frame are subjected to IOU matching through a matching module, and short-term association is achieved; cross-frame correlation is calculated for unmatched detection through a history backtracking module, and long-term correlation is achieved; and finally, combining the two association results to obtain a complete tracking result. According to the method, the tracking precision can be remarkably improved in a complex scene, and higher robustness is achieved in the aspect of target identity maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a multi-target tracking method based on long-term and short-term trajectory association. Background Art

[0002] Multi-object tracking (MOT) is an important research area in computer vision. However, in complex scenes, factors such as target similarity and long-term occlusion often lead to problems such as track breakage and identity switching. Although the tracking-by-detection paradigm has made significant progress by separating detection and association tasks, its reliance on heuristic matching rules (such as Intersection over Union (IoU) and Re-ID features) and post-processing strategies (such as trajectory interpolation) prevents joint optimization of detection and association tasks. It also struggles to model long time series to address long-term occlusion, target disappearance, and identity switching in complex motion scenarios.

[0003] In recent years, end-to-end Transformer-based methods have significantly improved the robustness of short-term dependencies by implicitly modeling target motion and appearance changes through iteratively updating the track query. However, these methods still face two major bottlenecks. The first is the optimization conflict between detection and tracking tasks. Existing methods jointly optimize the detection query and the track query, but detection performance is limited by the end-to-end Transformer framework, resulting in high missed detections and false detection rates. While some improved solutions have attempted to alleviate the detection problem by introducing pre-trained detectors to generate proposal queries, the implicit decoupling of detection and tracking still cannot completely eliminate the gradient interference between tasks. The second is the inadequate modeling of long-term dependencies. Existing methods primarily rely on local information from adjacent frames and lack a global understanding of the multi-frame historical context. For example, some methods fuse the trajectory features of the current frame with the previous frame through temporal aggregation mechanisms to preserve short-term trajectory information. However, their online processing mechanisms and exit strategies limit the ability to track objects that are occluded or disappear for a long time. Although other methods provide position priors through detectors, they do not design a recovery mechanism for long-term lost targets, which makes it difficult to solve the problem of trajectory interruption in occlusion scenarios.

[0004] To address the above issues, a series of solutions have been proposed in this field through multi-dimensional innovation: Patent CN116681729A proposes two-level correlation matching and hypothetical box tracking. This method generates target prediction boxes through trajectory prediction. If the initial matching fails, a secondary matching based on the neighboring edge graph is initiated. This method, combined with a continuous tracking mechanism for the hypothetical box, addresses target loss under severe occlusion. This method has been validated for its occlusion resistance on the MOTChallenge dataset, but does not address the joint optimization of detection and tracking. Patent CN116758110A designs an IoU-ReID fusion matching and observation amplification method for complex motion scenes. By adaptively weighting appearance and motion features and expanding the prediction box size, it improves the matching probability of occluded targets. Its trajectory state management module further optimizes the stability of long-term tracking, but it is still insufficient for long-term modeling of nonlinear motion. Patent CN117252908A proposes a multi-scale dual attention network, which tracks the target area through spatial and channel attention focus, and designs a temporal aggregation network to fuse historical features to suppress occlusion noise. Its adaptive weight mechanism dynamically adjusts the feature enhancement strength, effectively improving identity consistency in complex interactive scenarios, but the recovery of long-lost targets still relies on external detectors; Patent CN119048563A proposes a confidence-based selective feature update module (CASFUM), which fuses trajectory confidence and detection confidence through an adaptive EMA mechanism, filters the noise characteristics of the trajectory in the occluded area, and combines the AAAM correction module (AAAM) to dynamically select historical feature templates, significantly improving the association robustness under dense occlusion. However, its strong detection dependence and insufficient long-term temporal modeling still restrict the tracking performance in complex dynamic scenes. Patent CN117173221A innovatively introduces a authenticity grading strategy, using presence scores to differentiate high- and low-confidence detection targets. It also employs the Restoration of Overlap (RIoU) and occluded person re-identification modules to enhance feature extraction and matching accuracy for occluded targets. This approach significantly reduces false detections, but does not fully exploit cross-frame temporal information.

[0005] In summary, current multi-target tracking methods in complex scenarios still face core bottlenecks such as insufficient coordinated optimization of detection and association tasks and limited ability to model long-term context. These bottlenecks lead to limited detection accuracy, poor trajectory continuity, and frequent identity switching. A method that can uniformly model short-term motion prediction and long-term trajectory recovery is needed to improve tracking robustness in complex dynamic environments. Summary of the Invention

[0006] In response to the above technical problems, the technical solution adopted by the present invention is: a multi-target tracking method based on the association of long-term and short-term trajectories, which solves the problems of missed detection, false detection, identity switching, and trajectory breakage in occlusion scenarios in the existing technology by collaboratively optimizing short-term motion prediction and long-term trajectory recovery, and realizes robust tracking in complex scenarios.

[0007] This paper proposes a multi-target tracking method based on long-term and short-term trajectory association, which includes the following core modules: 1. Detection module: The detection module uses the YOLOX detector to generate high-precision detection boxes in real time and output detection queries; 2. Motion Prediction Module: The motion prediction module uses the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. The motion information includes the center coordinates (x, y), height h, width w, confidence s, and appearance information. 3. Track Query Memory Module (TQMM): The TQMM module contains a spatiotemporal enhancement module, a query memory module, and an update module. The TQMM module uses the internal spatiotemporal enhancement module to enhance the trajectory information of the current frame. The internal query memory module and the update module store the track information of the tracked target. The update module and the clearing mechanism continuously update the information to provide the latest track information of lost targets for long-term association. 4. Matching module: The matching module performs IOU matching between the prediction box generated by the motion prediction module and the detection box generated by the detection module for each frame. The target of a successful match achieves short-term association. 5. History Backtracking Module: The history backtracking module calculates the correlation between the detection targets that have not been successfully matched in the short-term association and the trajectories in the lost state stored in the TQMM module, and performs cross-frame association on the unmatched detections. The targets that are successfully associated are then long-term associated.

[0008] The present invention proposes a multi-target tracking method based on long-term and short-term trajectory association, which specifically includes the following steps: 1) First, the YOLOX detector detects objects in the input video stream, generates high-precision detection boxes in real time, and provides detection queries for long-term and short-term associations; 2) Using the MOTR-ConvNext network, the motion information of the tracked target is predicted through dynamically updated trajectory queries, detection queries, and image features. Meanwhile, the Track Query Memory Module (TQMM) enhances, stores, and updates the trajectory information through its internal spatiotemporal enhancement module, query memory module, and update module. 3) The matching module performs IOU matching between the predicted box and the detection box of each frame to achieve short-term association; 4) The history backtracking module calculates the correlation between the mismatched detections and the memory traces in the lost state to achieve long-term association; 5) Integrate the results of short-term association and long-term association to obtain the complete tracking result of each frame.

[0009] Wherein the step 1) includes (taking the tth frame as an example): 1-1) A video stream of a given input of t frames is passed through the YOLOX detector to obtain the detection results of the t-1th frame and the detection results of the tth frame. The YOLOX detector uses the YOLOX weights provided by the official dataset. The detection results of each frame are represented by W bounding boxes, each of which is represented by the center coordinates (x, y), height h, width w, and confidence s. 1-2) By introducing shared queries, the bounding boxes are generated into a set of proposal queries. The proposal queries will replace the detection queries in MOTR-ConvNext to detect new targets. In fact, the generated proposal queries are the detection queries provided for long-term and short-term associations.

[0010] Wherein said step 2) includes (taking the tth frame as an example): 2-1) The MOTR-ConvNext network obtains the trajectory query of frame t-1 through the MOTR-ConvNext network of frame t-2; 2-2) The concatenated trajectory query of frame t-1 and the detection query of frame t-1, along with the image features of frame t-1, are input into the decoder of MOTR-ConvNext; 2-3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in the t-th frame, and perform motion prediction on the tracked target. The hidden states are also fed back to the Query Interaction Module (QIM) in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame (the t-th frame) based on the hidden state of the previous frame. 2-4) The TQMM module first aggregates the trajectory information of the historical frames and the current frame through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. It then stores the trajectory query information of all targets that have appeared through the internal query memory module combined with the internal update module. The trajectory information in the TQMM is continuously updated through the update module and the internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association.

[0011] Wherein said steps 2-4) include: 2-4-1) The spatiotemporal enhancement module aggregates the previous T S The trajectory information of the frame and the current frame is used to enhance the trajectory information of the current frame. The spatiotemporal enhancement module consists of 4 layers, each of which consists of 4 main parts: self-attention layer, matrix multiplication and Softmax layer, weighted summation layer and feedforward neural network layer. The present invention converts the timestamp into position embedding to further refine the processing capability of the model, and sets a memory buffer to store the trajectory query information of multiple historical frames in the spatiotemporal enhancement module. The buffer adopts the first-in-first-out (FIFO) data structure to achieve a maximum storage of T SFrame time length, each frame can store up to M object information; 2-4-2) The current frame trajectory query fused with context information is similarly estimated to the memory trajectory query stored in the query memory module, and the current trajectory query is stored through the storage mechanism; the historical trajectory queries of all tracked objects that have appeared are stored in the query memory module, and the memory module maintains the maximum length of the memory bank through the sliding window mechanism, and the window length is T w Frame, saves at most M object information; 2-4-3) The trajectory queries that have not been updated for a long time are cleared through the clearing mechanism, and the trajectory information in TQMM is continuously updated.

[0012] Wherein said step 2-4-1) comprises: 2-4-1-1) First, query the trajectory of the current frame (frame t) By updating the self-attention, the information representation of itself is enhanced. In this step, the dimension of each query in each frame is D, and the t-th frame has N t Trajectory query, front T S Frames (historical frames) are ···N t-1 trajectory query, the trajectory query of the current frame is Front T S The trajectory queries of the frames are Then it is used as Q through linear transformation, and the historical trajectory query in the memory buffer is used as both K and V through linear transformation: Where Q, K, and V are query, key, and value matrices respectively. For historical trajectory query, P t and P h is the time coding matrix, W q ,W k and W v is the linear transformation matrix, N is the number of trajectories, and T S +1 The maximum number of targets in a frame. For frames with fewer than N targets, fill them up to N with zero vectors. 2-4-1-2) Perform matrix product on Q and K, and then normalize them through the Softmax layer to obtain the similarity matrix where d k is the dimension of the key vector, QK T Represents the dot product between the query and the key. This step is used to measure the similarity between the query and all keys, and then divided by To scale the dot product result, where M∈R N×N Is a mask matrix used to ignore the influence of padding items; 2-4-1-3) Use the similarity matrix to weight the historical trajectory query. Multiply the similarity matrix A by V to obtain the historical trajectory query that integrates the current frame information, that is, 2-4-1-4) Add the current trajectory query to the current historical trajectory query to generate a current trajectory query that incorporates historical frame information: 2-4-1-5) Finally, position feedforward network (FFN) and layer normalization are used for further adjustment. The feedforward neural network (FFN) layer consists of two linear transformations and a ReLU activation function. First, the weighted summation result is mapped to a high-dimensional space through the first linear transformation (FC layer), and then the nonlinear characteristics are introduced through the ReLU activation function to enhance the model's expression ability. Finally, the high-dimensional features are restored to the target dimension (N×D) through the second linear transformation (FC layer); the layer normalization (LN layer) performs a normalization operation on the features after FFN processing to eliminate the data distribution differences between different layers to accelerate training convergence. After adjustment, the trajectory query of the t-th frame fused with context information is finally obtained. Where FC represents the linear projection layer, LN represents the layer normalization, and Re represents the ReLU activation function.

[0013] Wherein said step 3) comprises: 3-1) Calculate the IOU value between the predicted box and the detected box. IOU quantifies the degree of overlap by calculating the ratio of the intersection area of ​​the two bounding boxes to the union area. The formula is: IOU(A,B) = Area(A∩B) / Area(A∪B), where A and B are the areas of the two bounding boxes, ∩ represents the intersection, and ∪ represents the union. The IOU value range is [0,1], and a larger value indicates a higher degree of overlap. 3-2) Set the threshold for judgment and set the IOU threshold α = 0.6 to determine whether the predicted box matches the detection box. If the IOU is greater than the threshold, it is considered that the two are matched successfully and the target achieves short-term association; otherwise, the two are not matched successfully and the detection requires long-term association.

[0014] Wherein said step 4) includes (taking the tth frame as an example): 4-1) Assume that there are d unmatched detections after the short-term association of the t-th frame, and query the corresponding detections Q dtu ∈R d×D Query Q with the memory trace stored in the TQMM module in the lost state trml ∈R t×D The correlation calculation is performed in (assuming that TQMM stores t memory traces in the lost state), and the correlation matrix A is obtained through the calculation module sim ∈[0, 1] t×d; 4-2) Compare the elements of each column or row of the correlation matrix and find the elements with high values ​​in each column or row, where the correlation matrix A sim Each element in represents the association probability of an unmatched detection target to another memory trajectory in a lost state, that is, the targets corresponding to two queries with high correlation are the same target, and the target with successful matching achieves long-term association. If the match is still unsuccessful, that is, the correlation between the detection query of the unmatched detection target and the trajectory queries of all lost targets stored in the memory module is low, then the detection target is considered to be a new target and is used for target prediction in the next frame and to generate a trajectory query for the new target.

[0015] Wherein said step 4-1) comprises: 4-1-1) First, the two queries are separated by a self-attention mechanism to capture the dependencies within the sequence and generate enhanced feature representations. The self-attention formula is as follows: Where Q, K and V are query, key and value matrices respectively, d k Is the dimension of the key, used to scale the attention weight; 4-1-2) Use padding and masking to ensure that the two input tensors are of the same size, while providing information to ignore invalid padding data, and obtain two trajectory queries of the same size by padding M is the number of trajectories after zero vector filling, which is also the maximum number of targets in two trajectory queries (assuming that the memory trajectory has the most targets), and D is the dimension of each query; 4-1-3) will lose the memory trajectory query After linear transformation, we get the query (Q), that is, Will detect the query After linear transformation, we get the key (K), namely Where W q and W k It is a linear transformation matrix; 4-1-4) Then we use the cross attention method to realize the correlation calculation, so that each element in a sequence can pay attention to all elements in another sequence, and obtain the degree of attention. First, we perform matrix product on Q and K, and then normalize them through the Softmax function to obtain the correlation matrix where M∈R M×M It is a mask matrix, which is still used to ignore the influence of the filling items, through the correlation matrix A cor ∈[0,1] can get the detection query Q of the unmatched target du Each trace in the memory trace query Q is in the lost state trml The correlation of each trajectory in .

[0016] The beneficial effects of adopting the above technical solution are: 1. Combining the YOLOX detector with the MOTR-ConvNext model, the dynamic updated trajectory query and detection query jointly model the motion and appearance features between adjacent frames to optimize short-term motion prediction. The ConvNeXt backbone network enhances the contextual awareness of occluded objects by expanding the receptive field and integrating hierarchical features. 2. Design a trajectory query memory module. This module stores and updates historical trajectory information based on a spatiotemporal enhancement mechanism and an exponential moving average (EMA) strategy. It achieves robust matching of long-term lost targets through multi-frame context perception. 3. A historical backtracking module is introduced to perform global cross-frame association on unmatched detections, thus achieving recovery and identity preservation of long-term lost targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can derive other drawings based on these drawings without inventive effort. Figure 1 This is a general framework diagram of a multi-target tracking method based on long-term and short-term trajectory association provided by the present invention; Figure 2 A flowchart of a multi-target tracking method based on long-term and short-term trajectory association provided by the present invention; Figure 3 It is a structural diagram of the MOTR-ConvNext network provided by the present invention; Figure 4 This is a structural diagram of the TQMM module provided by the present invention; Figure 5 This is a visualization result diagram of a multi-target tracking method based on long-term and short-term trajectory association on the MOT17 dataset provided by the present invention; Figure 6 This is a visualization result diagram of a multi-target tracking method based on long-term and short-term trajectory association provided by the present invention on the DanceTrack dataset. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0019] Figure 1This figure shows the overall framework of a multi-target tracking method based on long-term and short-term trajectory association, which is provided by the present invention. This method achieves this goal by performing target detection on each video frame and then performing short-term and long-term association. 1. Target detection, which is performed by the detection module, uses the YOLOX detector to generate high-precision detection boxes in real time and provide detection queries for long-term and short-term association. 2. Short-term association. This step includes a motion prediction module, a track query memory module (TQMM), and a matching module. The motion prediction module uses the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated track queries, detection queries, and image features. The MOTR-ConvNext network iteratively predicts the target trajectory sequence. The motion information includes the center coordinates (x, y), height h, width w, confidence s, and appearance information. The TQMM module enhances the current frame trajectory information through the spatiotemporal enhancement module, stores the track information of the tracked target through the query memory module combined with the update module, and continuously updates the information through the update module and the internal clearing mechanism to provide the latest track information of the lost target for long-term association. The matching module performs IOU matching on the prediction box generated by the motion prediction module for each frame with the detection box generated by the detection module. The target that is successfully matched achieves short-term association. 3. Long-term association, long-term association is completed by the historical backtracking module, which calculates the correlation between the unsuccessful matching detections in the short-term association and the lost trajectories stored in the TQMM module and generates the correlation matrix A sim ∈[0, 1] t×d ,The unmatched detections are associated across frames through the correlation scores, and the goal of successful association is to achieve long-term association.

[0020] Figure 2 This is a flow chart of a multi-target tracking method based on long-term and short-term trajectory association provided by the present invention. The multi-target tracking method based on long-term and short-term trajectory association can be implemented by combining the above multi-target tracking method framework. The specific steps are as follows (taking the tth frame as an example): Step S201 first performs object detection on the input video stream using the YOLOX detector, generates high-precision detection frames in real time, and provides detection queries for long-term and short-term association. The specific steps include: (1) The video stream of the given input t frame is passed through the YOLOX detector to obtain the detection result of the t-1 frame And the detection results of frame t The YOLOX detector uses the YOLOX weights provided by the official dataset. The detection results of each frame are represented by W bounding boxes, each of which is represented by the center coordinates (x, y), height h, width w, and confidence s. (2) By introducing shared queries, the bounding box generates a set of proposal queries, which will replace the detection query Q in MOTR-ConvNext d ∈R L×D To detect new targets, L is the number of new targets, and D is the dimension of each trajectory. Figure 1 The detection query output by the target detection process is the detection query after being replaced by the proposed query. Step S202 uses the MOTR-ConvNext network to predict the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. At the same time, the TQMM module stores and updates the trajectory information through its internal query memory module and update module. The specific steps include: (1) The MOTR-ConvNext network obtains the trajectory query of the t-1 frame through the MOTR-ConvNext network of the t-2 frame (N t-1 is the number of trajectories in frame t-1); (2) Query the trajectory of frame t-1 and the detection query of frame t-1 (L t-1 is the number of new targets in frame t-1), the concatenated query and the image feature f of frame t-1 t-1 Input together into the decoder in MOTR-ConvNext; (3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in the t-th frame, and perform motion prediction on the tracked target. The hidden states are also fed back to the query interaction module (QIM) in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame (the t-th frame) based on the hidden state of the previous frame. (4) The TQMM module first aggregates the trajectory information of the historical frame and the current frame through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. Then, the internal query memory module is combined with the internal update module to store the trajectory query information of all targets that have appeared. The update module and the internal clearing mechanism are used to continuously update the trajectory information in the TQMM, providing the latest trajectory information of the lost targets for long-term association. Step S203 uses the matching module to perform IOU matching between the predicted frame and the detection frame of the tth frame to achieve short-term association. The specific steps include: 1) Calculate the IOU value between the predicted box and the detection box. IOU matching is the core algorithm in target detection and multi-target tracking. It is used to measure the degree of spatial overlap between two bounding boxes (BoundingBox) to determine whether they belong to the same target. IOU quantifies the degree of overlap by calculating the ratio of the intersection area to the union area of ​​the two bounding boxes. The formula is: IOU(A,B)=Area(A∩B) / Area(A∪B), where A and B are the areas of the two bounding boxes, ∩ represents the intersection, and ∪ represents the union. The value range of IOU is [0,1], and the larger the value, the higher the overlap; (2) Set a threshold for judgment. Set the IOU threshold α = 0.6 to determine whether the predicted box matches the detection box. If the IOU is greater than or equal to the threshold, it is considered that the two are matched successfully and the target achieves short-term association. Otherwise, the two are not matched successfully and the detection requires long-term association. Step S204 uses the history backtracking module to associate the unmatched detections with the lost trajectories for a long period of time. The specific steps include: (1) Assume that there are d unmatched detections after the short-term association of the t-th frame, and query the corresponding detections Q dtu ∈R d ×D , and the memory trace query Q stored in the TQMM module in the lost state trml ∈R t×D The correlation calculation is performed in (assuming that TQMM stores t memory traces in the lost state), D is the dimension of each trace, and the correlation matrix A is obtained through the calculation module sim ∈[0,1] t×d The correlation calculation process is as follows: ① First, the two queries are separated by a self-attention mechanism to capture the dependencies within the sequence and generate enhanced feature representations. The self-attention formula is as follows: Where Q, K and V are query, key and value matrices respectively. k is the dimension of the key, used to scale the attention weights; ② Use padding and masking to ensure that the two input tensors are of the same size, while providing information to ignore invalid padding data, and obtain two trajectory queries of the same size through padding M is the number of trajectories after zero vector filling, which is also the maximum number of targets in two trajectory queries (assuming that the memory trajectory has the most targets), and D is the dimension of each query; ③ The memory trajectory query will be lost After linear transformation, we get the query (Q), that is, Will detect the query After linear transformation, we get the key (K), namely Where W q and W k is the linear transformation matrix; ④ Then we use the cross attention method to realize the correlation calculation, so that each element in one sequence can pay attention to all elements in another sequence and obtain the degree of attention. First, we perform matrix product on Q and K, and then normalize them through the Softmax function to obtain the correlation matrix where M∈R M×M Is a mask matrix, still used to ignore the influence of the filling items. Through the correlation matrix A cor ∈[0,1], we can get the detection query Q of the unmatched target du Each trace in the memory trace query Q is in the lost state trml The correlation of each trajectory in ; (2) Compare the elements of each column or row of the correlation matrix and find the element with the highest value in each column or row. sim Each element in represents the probability of association of an unmatched detection target with another lost memory track. That is, the targets corresponding to two queries with high correlation are the same target, and the target that is successfully matched has achieved long-term association. If a match is still not successful, that is, the correlation between the unmatched detection and all the tracks of the lost targets stored in the memory module is low, then the detection target is considered a new target and used for target prediction in the next frame and generates a new target track query. Step S205 combines the results of short-term and long-term associations and outputs the complete tracking result of the t-th frame.

[0021] Figure 3 This is the structure diagram of the MOTR-ConvNext network provided by the present invention. The MOTR-ConvNext network iteratively processes each frame of video and predicts the motion information of the tracked target through dynamically updated trajectory queries, detection queries, and image features. The specific steps are as follows: Step 1: The video sequence will first be sent to the ConvNext backbone network (the present invention actually uses the ConvNext-Base network) for feature extraction. The network is based on a pure convolutional architecture, combining the global modeling capabilities of Transformer with the local perception characteristics of CNN. It stacks multi-layer convolution blocks in four stages (including depth-separable convolution, layer normalization, and GELU activation function) to extract multi-scale semantic features of the image layer by layer. The selected ConvNeXt-Base network is a medium-sized version of the series with approximately 88M parameters. The number of channels in the four stages is set to 128, 256, 512, and 1024, respectively, and the ratio of the number of modules in each stage is 1:1:3:1. This design enhances the ability to extract abstract semantic features while retaining local details, and is particularly suitable for feature encoding of high-resolution frames in video sequence processing. The extracted features are then input into the Deformable DETR encoder (Encoder, abbreviated as Enc). The core improvement of Enc is the introduction of a deformable attention mechanism, which reduces the computational complexity by sparsely sampling feature points at key positions while retaining the ability to perceive multi-scale targets. The encoder aggregates global context information through multi-layer self-attention and cross-attention modules, and combines sine and cosine position encoding to model spatial relationships, ultimately generating image features f = {f1,...,f i ,...,f N}, where f1 represents the feature of frame T1, and the video sequence is passed through the YOLOX detector to obtain the detection query The YOLOX detector is based on a lightweight anchor-free design. After extracting multi-level features through the CSPDarknet backbone network, the Feature Pyramid Network (FPN) fuses semantic information at different scales. The decoupling head then handles classification and regression tasks separately. The classification branch predicts the target category probability through multi-layer convolution, while the regression branch directly outputs the bounding box coordinates. The SimOTA label assignment strategy is used to optimize the matching of positive and negative samples, achieving efficient and accurate real-time target detection. Step 2: Query Interaction Module (QIM) queries based on the initial trajectory of the previous frame Generate trajectory query ( Generated by initialization), where the QIM module is a component designed based on the Transformer architecture. The structure of QIM includes an object entry and exit mechanism and a temporal aggregation network (TAN), which dynamically maintains the trajectory query set of the target in the video sequence and strengthens the modeling of long-term sequences. QIM dynamically filters the high-scoring trajectory queries from the initial trajectory query (hidden state) generated by the Transformer decoder through the object entry and exit mechanism, and introduces a temporal aggregation network to interact the hidden state of the current frame with the historical trajectory information of the previous frame through a multi-head self-attention mechanism, aggregating long-term temporal features to enhance the modeling ability of the target motion, and then generating the trajectory query for the next frame. Then the trajectory query Detection query corresponding to each frame Cascaded together, the query and feature f after cascading i (i∈[1,N]) are sent to the decoder (Decoder, abbreviated as Dec). Dec is a modular structure designed based on the Transformer architecture. It uses a multi-layer Transformer structure, including a self-attention mechanism (Self-Attention) and an encoder-decoder attention mechanism (Cross-Attention). The former captures the temporal correlation between the query and the feature by calculating the similarity between the query, key, and value; the latter uses the feature map extracted by the encoder as context information and dynamically locates the target area in the feature map through the attention weight. The decoder also contains a feedforward neural network (FFN) to perform nonlinear transformation and feature fusion on the information output by the attention to enhance the model's expression ability, and finally generates the initial trajectory query for the current frame. Initial trajectory query It is a hidden state, and its function is to predict the bounding box of the tracked object in the next frame and perform motion prediction on the tracked target; Step 3: Then pass Generate prediction results for the current frame Prediction results Including center coordinates (x, y), height h, width w and confidence s and other information, while the initial trajectory query Feedback to QIM, and then update through QIM Send it to the next frame. For the T1 frame (first frame), the trajectory query is initialized to empty, with only feature f1 and detection query is input into Dec to interact and generate the initial trajectory query of frame T1 Then generate the prediction of the current frame And the trajectory query of the next frame (frame T2)

[0022] Figure 4 This is a diagram of the TQMM module structure provided by the present invention. The TQMM module first enhances the current trajectory information through the internal spatiotemporal enhancement module, then stores the trajectory query information of all targets that have appeared through the internal query memory module and update module, and continuously updates the trajectory information in the TQMM through the update module and internal clearing mechanism. The specific steps are as follows: Step 1: The spatiotemporal enhancement module aggregates the previous T S The spatiotemporal enhancement module is composed of 4 layers, each of which consists of 4 main parts: self-attention layer, matrix multiplication and Softmax layer, weighted summation layer and feedforward neural network layer. The present invention converts timestamps into position embeddings to further refine the processing capability of the model, and sets a memory buffer to store the trajectory query information of multiple historical frames in the spatiotemporal enhancement module. The buffer can store up to T S Frame time length, each frame can store up to M object information. The buffer is implemented using a first-in-first-out (FIFO) data structure. As new frame information is added, the oldest data in the queue is discarded; Step 1.1: First, query the trajectory of the current frame (frame t) The self-attention layer is updated to enhance its own information representation. The self-attention mechanism is a mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. This mechanism allows the model to dynamically adjust the degree of attention to each element when processing sequence data, thereby capturing the complex dependencies within the sequence and enhancing information representation. In this step, the dimension of each query in each frame is D, and the tth frame has N t Trajectory query, front T S Frames (historical frames) are ···N t-1 trajectory query, the trajectory query of the current frame is Front T S The trajectory queries of frames (historical frames) are Then it is used as Q through linear transformation, and the historical trajectory query in the memory buffer is used as both K and V through linear transformation: Where Q, K, and V are query, key, and value matrices respectively. For historical trajectory query, P t and P h is the time coding matrix, W q ,W k and Wv is the linear transformation matrix, N is the number of trajectories, and T S +1 Maximum number of targets in a frame. For frames with fewer than N targets, fill with zero vectors up to N; Step 1.2: Perform matrix product on Q and K, and then normalize them through the Softmax layer to obtain the similarity matrix where d k is the dimension of the key vector, QK T Represents the dot product between the query and the key. This step is used to measure the similarity between the query and all keys, and then divided by To scale the dot product result, where M∈R N×N Is a mask matrix used to ignore the influence of padding items; Step 1.3: Use the similarity matrix to weight the historical trajectory query. Multiply the similarity matrix A by V to obtain the historical trajectory query that integrates the current frame information, that is, Step 1.4: Add the current trajectory query to the current historical trajectory query to generate the current trajectory query that incorporates the historical frame information: Step 1.5: Finally, further adjustments are made using the position feedforward network (FFN) and layer normalization. The feedforward neural network (FFN) layer consists of two linear transformations and a ReLU activation function. First, the weighted summation result is mapped to a high-dimensional space through the first linear transformation (FC layer), and then the nonlinear characteristics are introduced through the ReLU activation function to enhance the model's expressiveness. Finally, the high-dimensional features are restored to the target dimension (N×D) through the second linear transformation (FC layer); the layer normalization (LN layer) normalizes the features after FFN processing to eliminate the data distribution differences between different layers to accelerate training convergence. After adjustment, the trajectory query of the t-th frame fused with context information is finally obtained. Where FC represents the linear projection layer, LN represents the layer normalization, and Re represents the ReLU activation function; Step 2: Estimating the similarity between the current frame trajectory query that integrates the context information and the memory trajectory query stored in the query memory module, and then storing the current trajectory query information through the storage mechanism. The historical trajectory queries of all tracked objects that have appeared are stored in the query memory module. The memory module maintains the maximum length of the memory bank through the sliding window mechanism, and the window length is T w Frame, saves at most M object information; Step 2.1: First, use cosine similarity to match each trajectory in the current trajectory query with each trajectory in the memory trajectory query to obtain a similarity score. Matrix operations are used to calculate the similarity between all trajectory queries and memory trajectory query entries at once, thereby improving the calculation speed. Finally, a similarity matrix S∈R is obtained. N×MThe cosine similarity formula is Among them S ij Represents the current trajectory query Q′ trc ∈R N×D The i-th trajectory c i and memory trace query Q trm ∈R M×D The jth trajectory m in j The cosine similarity of M and N is the number of trajectories, and D is the vector dimension; Step 2.2: Query the memory module update, by setting the similarity threshold S∈[0,1] to determine whether the target of the current frame is a new target, and then update the memory trajectory query in the query memory module according to the update mechanism. If the target similarity in the current frame is greater than or equal to the similarity threshold S, then the target is an old target (the tracked target), and the trajectory information of the target is smoothly overwritten with the corresponding information in the memory trajectory query, and the unupdated counter is initialized at the same time (counting the number of unupdated frames of target information); since direct coverage may cause feature information mutation, the exponential moving average (EMA) is used to smooth the information update, EMA t =αx t +(1-α)EMA t-1 , where EMA t is the exponential moving average of time t, x t is the actual data value at time t, EMA t-1 is the exponential moving average at time t-1. α is a smoothing factor hyperparameter, usually α∈(0,1). If the target similarity in the current frame is less than the similarity threshold S, the target is considered a new target (a newly appeared target), and the trajectory information corresponding to the target is stored in the query memory module, and the unupdated counter is initialized at the same time; Step 3: Clear the trajectory queries that have not been updated for a long time through the clearing mechanism, and then continuously update the trajectory information in TQMM.

[0023] For the clearing mechanism, the trajectory query in the query memory module is divided into three states: survival, loss, and inactivation. The frame number threshold F∈(1,T W ) to determine the status of the memory trace query in the query memory module. The unupdated counter is updated frame by frame. If the value of the unupdated counter of a target in the memory trace query is 0, the target is alive and is recorded as q trma If the value of the unupdated counter of a target in the memory trace query is greater than 1 and less than or equal to F, the target is lost and recorded as q trml If the value of the unupdated counter of a target in the memory trajectory query is greater than F, the target is deactivated and the query memory module deletes the trajectory information of the target.

[0024] To achieve effective training for multi-object tracking, we design a comprehensive loss function that takes both short-term and long-term associations into account.

[0025] For short-term association, the loss function of the t-th frame is expressed as in, is the classification loss in short-term associations, calculated using focal loss; is the L1 loss in short-term association; It is the generalized intersection-over-union loss in short-term associations. and Is the corresponding weight coefficient. For long-term association, the present invention takes into account the correlation matrix obtained by the historical backtracking module as A cor ∈[0,1], the loss function of long-term association is defined as (Assume that there are d mismatched detections after the short-term correlation of the t-th frame, and the TQMM stores t memory traces in the lost state.) Here The dissimilarity between the detection query that does not match the detection target and the query with lost memory trace is expressed. The loss of long-term association is obtained by summing up all the dissimilarities.

[0026] Finally, the total loss function of this method is the weighted sum of short-term association loss and long-term association loss where ω s and ω l are the weights of the short-term association loss and the long-term association loss, respectively, used to balance the importance of the two associations in training. Through loss function design, this method can simultaneously optimize the performance of short-term and long-term associations during training, thereby improving the overall effect of multi-target tracking.

[0027] To fully validate the performance of this method, we conducted experiments on two mainstream multi-object tracking datasets, MOT17 and DanceTrack, compared to state-of-the-art methods. Table 1 shows the performance comparison of this method with other state-of-the-art methods on the DanceTrack test set (↑ indicates higher performance). It should be noted that the experimental data for other state-of-the-art methods are cited from their original papers.

[0028] Table 1 Comparison with existing methods on the DanceTrack dataset

[0029] Table 1 shows that our method's IDF1 improves by 1.5 over MOTRv3 (72.3) and by 2.1 over MOTRv2 (71.7), validating the effectiveness of the long-term association module. MOTRv3 is an end-to-end Transformer-based method proposed in the paper "MOTRv3: Release-Fetch Supervision for End-to-End Multi-Object Tracking," and MOTRv2 is an end-to-end Transformer-based method proposed in the paper "MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors." We attribute this to the TQMM module dynamically maintaining the memory through EMA smoothing updates and sliding window cleanup, and restoring the identity of the target through the history backtracking module when the target reappears. Our method's AssA (60.1) is higher than MOTRv3's (59.3), indicating that its trajectory ID consistency over time is more robust. We attribute this to the ConvNeXt backbone's hierarchical feature enhancement for modeling complex motion, combined with the TQMM's spatiotemporal enhancement module, which optimizes trajectory continuity through multi-frame attention fusion. Our method's HOTA is 0.4 higher than MOTRv2 and slightly lower than MOTRv3 by 0.1, but its association accuracy (60.1) significantly outperforms both MOTRv3 (59.3) and MOTRv2 (59.0). Regarding HOTA's combined detection (DetA) and association (AssA) performance, our method's long-term association module reduces ID switching by restoring historical trajectories (as evidenced by an improvement in IDF1), but the introduced trajectory noise leads to a slight decrease in DetA, which in turn leads to a slight decrease in HOTA. Our method's MOTA (91.2) is slightly lower than MOTRv2 (91.9). We speculate that MOTA focuses more on detection and short-term tracking, neglecting trajectory continuity, while our method has a significant advantage in long-term association, resulting in its slightly lower MOTA.

[0030] Table 2 shows the performance comparison of this method and other SOTA methods on the MOT17 test set (↑ / ↓ indicates that the higher / lower the index, the better). It should be noted that the experimental data of other SOTA methods are all cited from their original papers.

[0031] Table 2 Comparison with existing methods on the MOT17 dataset

[0032] Table 2 shows that our method outperforms MOTRv3, with a HOTA improvement of 2.7 and an IDF1 improvement of 4.8. Our method's IDS is 33% lower than MOTRv3, indicating that the resulting trajectories are continuous and robust. Our method also performs better than MOTRv2. However, similar to the results on the DanceTrack test set, our method's MOTA is still slightly lower than MOTRv2. We speculate that this is due to our method's significant advantage in long-term associations. Furthermore, we can see that CNN-based methods (such as ByteTrack and OC-SORT) perform well on the MOT17 dataset, although they perform worse than our method on DanceTrack. We infer that this is because the object motion trajectories in MOT17 are simple, so by combining a powerful object detector such as YOLOX with the designed post-processing rules, we can effectively track objects in MOT17.

[0033] To demonstrate the tracking effect of the patented method, the visualization results of the patented algorithm and the classic MOTRv2 algorithm on the MOT17 dataset and DanceTrack dataset are shown.

[0034] On the MOT17 dataset, such as Figure 5 As shown in (a), pedestrian No. 41 is blocked by MOTRv2 at frame 234. At frame 267, pedestrian No. 41 appears again. However, since MOTRv2 relies on short-term matching and lacks long-term association, the ID of pedestrian No. 41 changes from 41 to 72. At the same time, pedestrian No. 72 is blocked. When pedestrian No. 72 appears again at frame 365, its ID changes from 72 to 126. Figure 5 (b) It can be clearly seen that the IDs of all pedestrians in each frame of this method have not changed.

[0035] On the DanceTrack dataset, Figure 6 As shown in (a), in MOTRv2, dancer No. 4 is blocked at frame 60. At frame 60, dancer No. 4 appears again, and the original dancer No. 4's ID changes from 4 to 6. At frame 115, occlusion occurs again. At frame 174, when all dancers appear again, the original dancer No. 1's ID changes from 1 to 7, the original dancer No. 3's ID changes from 2 to 8, and the original dancer No. 5's ID changes from 5 to 9. Figure 6 (b) It can be clearly seen that the IDs of all dancers in each frame of this method have not changed.

[0036] Through the above visualization and analysis, we can conclude that this method significantly reduces the ID Switch phenomenon in complex scenes (such as DanceTrack and MOT17) by associating long-term and short-term trajectories, indicating that the model has significant advantages in complex motion modeling and identity preservation.

[0037] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also encompass other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A multi-target tracking method based on long-term and short-term trajectory association, characterized in that: include: 1) First, the YOLOX detector detects objects in the input video stream, generates high-precision detection boxes in real time, and provides detection queries for long-term and short-term associations; 2) Using the MOTR-ConvNext network, the motion information of the tracked target is predicted through dynamically updated trajectory queries, detection queries, and image features. Meanwhile, the Track Query Memory Module (TQMM) enhances, stores, and updates the trajectory information through its internal spatiotemporal enhancement module, query memory module, and update module. 3) The matching module performs IOU matching between the predicted box and the detection box of each frame to achieve short-term association; 4) The history backtracking module calculates the correlation between the mismatched detections and the memory traces in the lost state to achieve long-term association; 5) Integrate the results of short-term association and long-term association to obtain the complete tracking result of each frame.

2. The multi-target tracking method based on long-term and short-term trajectory association according to claim 1 is characterized in that: The step 1) includes (taking the tth frame as an example): 1-1) Given an input video stream of t frames, pass it through the YOLOX detector to obtain the detection results of the t-1th frame and the detection results of the tth frame. The YOLOX detector uses the YOLOX weights provided by the official dataset. The detection results of each frame are represented by W bounding boxes, each of which is represented by the center coordinates (x, y), height h, width w, and confidence s. 1-2) By introducing shared queries, the bounding box generates a set of proposal queries. The proposal queries will replace the detection queries in MOTR-ConvNext to detect new targets. In fact, the generated proposal queries are the detection queries provided for long-term and short-term associations.

3. The multi-target tracking method based on long-term and short-term trajectory association according to claim 1 is characterized in that: The step 2) includes (taking the tth frame as an example): 2-1) The MOTR-ConvNext network obtains the trajectory query of the t-1 frame through the MOTR-ConvNext network of the t-2 frame; 2-2) The concatenated query of the trajectory query of the t-1th frame and the detection query of the t-1th frame, along with the image features of the t-1th frame, are input into the decoder in MOTR-ConvNext; 2-3) In the decoder, these queries interact with image features to generate hidden states, predict the bounding box of the tracked object in the t-th frame, and perform motion prediction on the tracked target. The hidden states are also fed back to the Query Interaction Module (QIM) in the MOTR-ConvNext model. The QIM module generates the trajectory query for the next frame (the t-th frame) based on the hidden state of the previous frame. 2-4) The TQMM module first aggregates the trajectory information of the historical frames and the current frame through the spatiotemporal enhancement module to enhance the trajectory information of the current frame. It then stores the trajectory query information of all targets that have appeared through the internal query memory module combined with the internal update module. The trajectory information in the TQMM is continuously updated through the update module and the internal clearing mechanism to provide the latest trajectory information of lost targets for long-term association.

4. The multi-target tracking method based on long-term and short-term trajectory association according to claim 3 is characterized in that: The steps 2-4) include: 2-4-1) The spatiotemporal enhancement module aggregates the previous T S The trajectory information of the frame and the current frame is used to enhance the trajectory information of the current frame. The spatiotemporal enhancement module consists of 4 layers, each of which consists of 4 main parts: self-attention layer, matrix multiplication and Softmax layer, weighted summation layer and feedforward neural network layer. The present invention converts timestamps into position embeddings to further refine the processing capability of the model. At the same time, a memory buffer is set to store the trajectory query information of multiple historical frames in the spatiotemporal enhancement module. The buffer adopts the first-in-first-out (FIFO) data structure to achieve a maximum storage of T S Frame time length, each frame can store up to M object information; 2-4-2) The current frame trajectory query fused with context information is similarly estimated to the memory trajectory query stored in the query memory module, and the current trajectory query is stored through the storage mechanism; the historical trajectory queries of all tracked objects that have appeared are stored in the query memory module, and the memory module maintains the maximum length of the memory bank through the sliding window mechanism, and the window length is T w Frame, saves at most M object information; 2-4-3) The trajectory queries that have not been updated for a long time are cleared through the clearing mechanism, and the trajectory information in TQMM is continuously updated.

5. The multi-target tracking method based on long-term and short-term trajectory association according to claim 4 is characterized in that: The step 2-4-1) includes: 2-4-1-1) First, query the trajectory of the current frame (frame t) By updating the self-attention, the information representation of itself is enhanced. In this step, the dimension of each query in each frame is D, and the t-th frame has N t Trajectory query, front T S Frames (historical frames) are trajectory query, the trajectory query of the current frame is Front T S The trajectory queries of the frames are Then it is used as Q through linear transformation, and the historical trajectory query in the memory buffer is used as both K and V through linear transformation: Where Q, K, and V are query, key, and value matrices respectively. For historical trajectory query, P t and P h is the time coding matrix, W q ,W k and W v is the linear transformation matrix, N is the number of trajectories, and T S +1 The maximum number of targets in a frame. For frames with fewer than N targets, fill them up to N with zero vectors. 2-4-1-2) Perform matrix product on Q and K, and then normalize them through the Softmax layer to obtain the similarity matrix where d k is the dimension of the key vector, QK T Represents the dot product between the query and the key. This step is used to measure the similarity between the query and all keys, and then divided by To scale the dot product result, where M∈R N×N Is a mask matrix used to ignore the influence of padding items; 2-4-1-3) Use the similarity matrix to weight the historical trajectory query. Multiply the similarity matrix A by V to obtain the historical trajectory query that integrates the current frame information, that is, 2-4-1-4) Add the current trajectory query to the current historical trajectory query to generate a current trajectory query that incorporates historical frame information: 2-4-1-5) Finally, position feedforward network (FFN) and layer normalization are used for further adjustment. The feedforward neural network (FFN) layer consists of two linear transformations and a ReLU activation function. First, the weighted summation result is mapped to a high-dimensional space through the first linear transformation (FC layer), and then the nonlinear characteristics are introduced through the ReLU activation function to enhance the model's expression ability. Finally, the high-dimensional features are restored to the target dimension (N×D) through the second linear transformation (FC layer); the layer normalization (LN layer) performs a normalization operation on the features after FFN processing to eliminate the data distribution differences between different layers to accelerate training convergence. After adjustment, the trajectory query of the t-th frame fused with context information is finally obtained. Where FC represents the linear projection layer, LN represents the layer normalization, and Re represents the ReLU activation function.

6. The multi-target tracking method based on long-term and short-term trajectory association according to claim 1 is characterized in that: The step 3) comprises: 3-1) Calculate the IOU value between the predicted box and the detected box. IOU quantifies the degree of overlap by calculating the ratio of the intersection area of ​​the two bounding boxes to the union area. The formula is: IOU(A,B) = Area(A∩B) / Area(A∪B), where A and B are the areas of the two bounding boxes, ∩ represents the intersection, and ∪ represents the union. The IOU value range is [0,1], and a larger value indicates a higher degree of overlap. 3-2) Set the threshold for judgment and set the IOU threshold α = 0.6 to determine whether the predicted box matches the detection box. If the IOU is greater than the threshold, it is considered that the two are matched successfully and the target achieves short-term association; otherwise, the two are not matched successfully and the detection requires long-term association.

7. The multi-target tracking method based on long-term and short-term trajectory association according to claim 1 is characterized in that: The step 4) includes (taking the tth frame as an example): 4-1) Assume that there are d unmatched detections after the short-term association of the t-th frame, and query the corresponding detections Q dtu ∈R d×D Query Q with the memory trace stored in the TQMM module in the lost state trml ∈R t×D The correlation calculation is performed in (assuming that TQMM stores t memory traces in the lost state), and the correlation matrix A is obtained through the calculation module sim ∈[0, 1] t×d ; 4-2) Compare the elements of each column or row of the correlation matrix and find the element with the highest value in each column or row. The correlation matrix A sim Each element in represents the association probability of an unmatched detection target to another memory trajectory in a lost state, that is, the targets corresponding to two queries with high correlation are the same target, and the target with successful matching achieves long-term association. If the match is still unsuccessful, that is, the correlation between the detection query of the unmatched detection target and the trajectory queries of all lost targets stored in the memory module is low, then the detection target is considered to be a new target and is used for target prediction in the next frame and to generate a trajectory query for the new target.

8. The multi-target tracking method based on long-term and short-term trajectory association according to claim 7 is characterized in that: The step 4-1) includes: 4-1-1) First, the two queries are separated by a self-attention mechanism to capture the dependencies within the sequence and generate enhanced feature representations. The self-attention formula is as follows: Where Q, K and V are query, key and value matrices respectively, d k is the dimension of the key, used to scale the attention weights; 4-1-2) Use padding and masking to ensure that the two input tensors are of the same size, while providing information to ignore invalid padding data, and obtain two trajectory queries of the same size through padding M is the number of trajectories after zero vector filling, which is also the maximum number of targets in two trajectory queries (assuming that the memory trajectory has the most targets), and D is the dimension of each query; 4-1-3) Query the lost memory trace After linear transformation, we get the query (Q), that is, Will detect the query After linear transformation, we get the key (K), namely Where W q and W k is the linear transformation matrix; 4-1-4) Then we use the cross-attention method to implement correlation calculation, so that each element in one sequence can pay attention to all elements in another sequence, and obtain the degree of attention. First, we perform matrix product on Q and K, and then normalize them through the Softmax function to obtain the correlation matrix where M∈R M×M It is a mask matrix, which is still used to ignore the influence of the filling items, through the correlation matrix A cor ∈[0,1], we can get the detection query Q of the unmatched target du Each trace in the memory trace query Q is in the lost state trml The correlation of each trajectory in .

Images