Transform multi-target tracking method based on relative position perception

Abstract

The invention relates to the technical field of computer vision, in particular to a Transform multi-target tracking method based on relative position perception. The method comprises the following steps: modeling a relative position relation of image features extracted by a target through a rotation attention module to obtain a tracking result; performing spatial clustering on the tracking result through a spatial grouping attention module to obtain a tracking query result; aggregating prediction results of multiple frames in a video sequence by adopting collective average loss (CAL), and optimizing time dependence of a rotation attention module and a space grouping attention module; and carrying out Transform multi-target tracking based on relative position sensing on the basis of the optimized module. The invention aims to enhance the perception ability of the model to the spatial relationship between the targets. According to the RP2Track, by introducing rotation position coding and a space grouping attention module, the capturing capability of the model on relative position information is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a Transformer multi-target tracking method based on relative position perception. Background Art

[0002] The goal of multi-target tracking algorithms in computer vision is to develop a method that can automatically identify and track multiple targets in a video or image. Its main goals include the following aspects:

[0003] 1. Object detection and recognition: The primary task of a multi-object tracking algorithm is to accurately detect and recognize multiple objects in an input image or video. This includes obtaining the object's location, bounding box, appearance features, and category information.

[0004] 2. Object tracking: The algorithm needs to be able to track the movement of the target in consecutive images or video frames. This involves the consistency and continuity of the target in time. The algorithm needs to predict and update the position of the target in the current frame based on the position information of the target in the previous frame.

[0005] 3. Robustness and real-time performance: Multi-target tracking algorithms must be robust and able to handle complex scenarios such as lighting changes, background clutter, and target deformation. Furthermore, they must offer real-time performance to efficiently track targets in live video streams.

[0006] The purpose of studying multi-target tracking algorithms is to improve the computer's ability to understand and analyze multiple targets in videos or images, and to provide more accurate and efficient target detection and tracking technologies for various application fields (such as video surveillance, autonomous driving, intelligent transportation, etc.).

[0007] Most existing efficient multi-target tracking algorithms fall into two modes: TBD (tracking by detection) and JDT (joint detection and tracking). The TBD mode first detects the target and then correlates its information over time. Mainstream TBD algorithms include SORT, DeepSORT, and ByteTrack. The JDT mode, on the other hand, simultaneously detects, extracts, and correlates target identity information within a single model. Mainstream JDT algorithms include JDE, TraDes, and MOTR.

[0008] The Tracking by Detection (TBD) paradigm treats multi-object tracking as two independent tasks: object detection and object tracking. First, an efficient detector is used to generate high-quality detection boxes, which are then processed by an association model, taking into account both appearance and motion cues. The association process utilizes pairwise appearance re-identification (Re-ID) similarities for appearance matching and employs various heuristics such as Intersection over Union (IoU) or Kalman filtering for motion-based association. Although the TBD paradigm has been widely used for multi-object tracking, its two-stage framework hinders end-to-end optimization during training.

[0009] In contrast, the Joint Detection and Tracking (JDT) paradigm aims to address this limitation by achieving detection and association of multiple objects in a single stage. However, JDT-based methods, such as MOTR and TraDeS, face optimization challenges due to the inherent competition in appearance learning, where detection increases inter-class variance during learning, while ReID increases intra-class variance, leading to poor tracking performance.

[0010] In recent years, multi-object tracking (MOT) has made significant progress in computer vision and is widely used in fields such as video surveillance, autonomous driving, and robotic navigation. The core goal of MOT is to accurately detect and track multiple objects in a video sequence while maintaining detection accuracy and temporal consistency of object identities. However, MOT still faces many challenges due to factors such as occlusion, deformation, and illumination variations.

[0011] Deep learning-based methods have significantly improved the performance of MOT, particularly the introduction of the Transformer architecture. The Transformer's self-attention mechanism effectively captures global dependencies between different positions in a sequence, thereby improving the accuracy of object association and significantly enhancing MOT's performance in complex scenarios.

[0012] Currently, MOT methods based on the Transformer architecture primarily draw on object detection technology. However, directly applying object detection methods to MOT often overlooks the unique requirements of tracking tasks. Object detection focuses on identifying individual objects in an image and annotating their bounding boxes, while MOT requires not only object recognition but also ensuring temporal consistency across frames and maintaining robustness to multi-object interactions and dynamic changes.

[0013] However, existing Transformer methods often overlook the importance of relative position information in multi-target tracking. Relative position information, such as the relative distance, orientation, and motion pattern between targets, is crucial for maintaining identity consistency and resolving occlusions. Many Transformer methods rely primarily on absolute position information while insufficiently modeling relative position relationships, limiting performance in complex motion scenes.

[0014] For example, in highly occluded scenes, traditional methods can easily cause identity confusion or loss by only capturing local features of the target, thus affecting tracking stability and accuracy. While the Transformer method improves the ability to model global dependencies, it lacks attention to local fine-grained information, making it difficult to effectively handle occlusion.

[0015] Currently, the Transform-based MOT framework largely inherits the Transform model used for object detection tasks. In these frameworks, each target is considered as an independent entity. For the tth frame in a series of tracking videos, the model first inherits the detection query from the Transform framework used in object detection. And additionally introduced tracking query from the previous frame as well as the temporal aggregation network, enabling the model to learn target tracking in an adaptive end-to-end manner. Here, E represents the embedding of the target and B represents the position information. However, when the target query interacts with the features, these methods still rely on absolute sinusoidal position encoding and standard attention mechanisms. This approach has certain limitations, especially in scenarios involving frequent interactions between targets. At the same time, the attention data caliber in Transform leads to defects in local attention, making it difficult for current Transform-based MOT methods to accurately track targets during occlusion and interaction. In these cases, adjacent targets are often mistakenly identified as the same target or cannot be distinguished at all. Summary of the Invention

[0016] In order to solve the problem in the existing technology that in highly occluded scenes, traditional methods are prone to identity confusion or loss due to only capturing local features of the target, thereby affecting the stability and accuracy of tracking. Although the Transformer method improves the modeling ability of global dependencies, it does not pay enough attention to local fine-grained information and is difficult to effectively handle the technical problem of occlusion. The embodiment of the present invention provides a Transformer multi-target tracking method and device based on relative position perception. The technical solution is as follows:

[0017] On the one hand, a Transformer multi-target tracking method based on relative position perception is provided, characterized in that the method includes:

[0018] S1. Obtain the target to be detected, detect the target, and extract image features of the target;

[0019] S2, model the relative position relationship of the image features extracted from the target through the rotation attention module to obtain the tracking result;

[0020] S3. Perform spatial clustering on the tracking results through the spatial grouping attention module to obtain the tracking query results;

[0021] S4. Using collective average loss CAL, we aggregate the prediction results of multiple frames in the video sequence and optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; based on the optimized modules, we perform relative position-aware Transformer multi-target tracking.

[0022] Optionally, in S2, the tracking result is obtained by rotating the attention module and modeling the relative position relationship of the image features extracted from the target, including:

[0023] Scale the target's bounding box position information by a preset factor to generate a scaled position vector;

[0024] Based on the diagonal rotation matrix of the rotation position encoding building block, the position embedding is rotated to generate relative position encoding;

[0025] Combining relative position encoding with target embedding, we compute attention scores guided by relative spatial relationships by modifying the query and key tensors.

[0026] Optionally, the rotation is based on the diagonal rotation matrix of the rotation position encoding building block, and the position embedding is rotated to generate a relative position encoding, including:

[0027] The position information of the markers is embedded in complex space and the relative position relationship is captured through rotation transformation;

[0028] Calculate the rotation angle based on the position embedding dimension and generate a block diagonal rotation submatrix;

[0029] The rotation sub-matrices are arranged along the block diagonal to form a block diagonal rotation matrix to generate relative position codes.

[0030] Optionally, spatial clustering is performed on the tracking results through a spatial grouping attention module to obtain tracking query results, including:

[0031] Compute the pairwise Euclidean distance matrix of the target bounding box to generate spatial proximity information;

[0032] Based on the distance matrix, the targets are grouped into spatially coherent groups through an iterative merging process;

[0033] Construct an attention mask to restrict the self-attention computation within the group and update the target embedding in combination with the rotation position encoding.

[0034] Optionally, based on the distance matrix, targets are grouped into spatially coherent groups through an iterative merging process consisting of:

[0035] Initialize each target as an independent group, and iteratively merge the groups based on the minimum average inter-group distance until the number of groups does not exceed the preset threshold;

[0036] The attention mask is used to restrict the self-attention mechanism within a group, allowing only targets within the same group to interact.

[0037] Optionally, a collective average loss CAL, comprising:

[0038] Match the predicted bounding box of each frame with the real annotation and calculate the classification loss, localization loss and intersection-over-union loss;

[0039] The multi-frame losses are normalized by the total number of targets to generate the total loss function and obtain the collective average loss CAL.

[0040] Optionally, the rotational attention module and the spatial grouping attention module are integrated into the Transformer decoder for end-to-end multi-object tracking.

[0041] On the other hand, a Transformer multi-target tracking device based on relative position perception is provided, which is applied to a Transformer multi-target tracking method based on relative position perception, and includes:

[0042] The feature extraction module is used to obtain the target to be detected, detect the target, and extract image features of the target;

[0043] The rotation attention module is used to model the relative position relationship of the image features extracted from the target to obtain tracking results;

[0044] The spatial grouping attention module is used to perform spatial clustering on the tracking results to obtain the tracking query results.

[0045] An optimized integration module is used to aggregate the prediction results of multiple frames in a video sequence using the collective average loss CAL, and to optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; based on the optimized module, Transformer multi-target tracking based on relative position perception is performed.

[0046] Optionally, a dataset initialization module is used to guide the red team model to generate strategic multi-round confrontation prompts based on a multi-thinking guidance approach, combined with a pre-attack thinking dataset;

[0047] The pre-attack thinking dataset classifies attack strategies into four categories: intent reversal, problem decomposition, role-playing, and hybrid modes, and requires the red team model to output the strategic thinking process before generating attack questions.

[0048] On the other hand, a Transformer multi-target tracking device based on relative position perception is provided, and the Transformer multi-target tracking device based on relative position perception includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned Transformer multi-target tracking methods based on relative position perception is implemented.

[0049] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned Transformer multi-target tracking methods based on relative position perception.

[0050] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0051] In this embodiment of the present invention, rotational position embedding (RoPE) is introduced to the field of multi-target tracking (MOT) for the first time, and a new relative position encoding strategy specifically tailored for multi-target tracking scenarios is proposed. By incorporating a rotation-based position perception mechanism, this method significantly improves the model's sensitivity to the spatial relationships between targets, thereby enhancing tracking performance. A spatial grouping attention module is proposed, which enhances the focus on local features in crowded scenes by constructing a localized attention mechanism. This design effectively improves the model's ability to perceive occluded targets, and especially demonstrates excellent performance in complex interactive scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A flowchart of a Transformer multi-target tracking method based on relative position perception provided by an embodiment of the present invention;

[0054] Figure 2 Detailed flow chart of a Transformer multi-target tracking method based on relative position perception provided by an embodiment of the present invention

[0055] Figure 3 A block diagram of a Transformer multi-target tracking device based on relative position perception provided by an embodiment of the present invention;

[0056] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0058] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0059] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0061] The embodiment of the present invention provides a Transformer multi-target tracking method based on relative position perception, which can be implemented by a Transformer multi-target tracking device based on relative position perception, which can be a terminal or a server. Figure 1 The flowchart of the Transformer multi-target tracking method based on relative position perception is shown in Figure 1 As shown, the Transformer multi-target tracking method based on relative position perception proposed in the present invention can include the following steps:

[0062] S1. Obtain the target to be detected, detect the target, and extract image features of the target;

[0063] S2. The relative position relationship of the image features extracted from the target is modeled through the Rotation Attention Module (RAM) to obtain the tracking result.

[0064] In one feasible implementation, in S2, the tracking result is obtained by rotating the attention module and modeling the relative position relationship of the image features extracted from the target, including:

[0065] Scale the target's bounding box position information by a preset factor to generate a scaled position vector;

[0066] Based on the diagonal rotation matrix of the rotation position encoding building block, the position embedding is rotated to generate relative position encoding;

[0067] Combining relative position encoding with target embedding, we compute attention scores guided by relative spatial relationships by modifying the query and key tensors.

[0068] In one feasible implementation, the rotation is based on the diagonal rotation matrix of the rotation position encoding building block, the position embedding is rotated to generate the relative position encoding, including:

[0069] The position information of the markers is embedded in complex space and the relative position relationship is captured through rotation transformation;

[0070] Calculate the rotation angle based on the position embedding dimension and generate a block diagonal rotation submatrix;

[0071] The rotation sub-matrices are arranged along the block diagonal to form a block diagonal rotation matrix to generate relative position codes.

[0072] In one possible implementation, positional encoding is crucial for models like the Transformer to understand the order of words in a sequence in natural language processing (NLP). Traditional positional encoding relies on absolute positions, which can be limited when modeling the relative relationship between tokens. To address this issue, Rotary Position Embedding (RoPE) was introduced, which uses rotation transformations to better capture relative position information. The core idea is to embed the position information of tokens into a complex space. By applying the rotation operation, the relative position information between tokens remains unchanged, thereby improving the model's ability to understand the order of the sequence.

[0073] like Figure 2 As shown in Figure 1, the present invention applies the core concept of RoPE to the multi-target tracking task, aiming to process the tracked objects by combining the position information Bt = [x, y, w, h] of the tracked objects with the tracking embedding Et.

[0074] For the t-th frame, in order to ensure compatibility with the sine function used in the self-attention mechanism, the present invention first scales Bt = [x, y, w, h] by a factor of 2π to obtain the scaled position vector p:

[0075] p t =2πB t

[0076] Subsequently, the present invention introduces a temperature parameter τ to adjust the attenuation of position information in different dimensions. For each dimension k, this attenuation is defined as:

[0077]

[0078] Where f represents the number of position features.

[0079] The present invention transforms the position embedding by constructing a block diagonal rotation matrix Rn, thereby encoding the relative directions between objects. The construction process is as follows:

[0080] First, calculate the rotation angle in each dimension:

[0081]

[0082] Here, d represents the dimension of position embedding.

[0083] Next, based on the position embedding and the calculated angle, each 2×2 rotation submatrix Rn,k is constructed:

[0084]

[0085] where Rn,k is the position embedding of the kth dimension of the nth sample.

[0086] These 2×2 rotation sub-matrices are placed along the block diagonal of the larger rotation matrix Rn:

[0087]

[0088] This block diagonal structure enables the rotation matrix Rn to effectively apply a rotation transformation to the position embedding, encoding the relative orientation between objects. Hereafter, the present invention will use RoPE to represent the implementation process of the above-mentioned rotation position encoding. In the multi-target tracking task, the ability to track objects based on the relative spatial relationship between objects is crucial to obtaining accurate results. As shown in the figure, the present invention integrates RoPE into the self-attention mechanism in the multi-target tracking decoder and the spatial grouping attention module (Spatial Grounding Attention Module, SGAM). Specifically, the present invention modifies the query (Qrope) and key (Krope) tensors by applying a rotation matrix:

[0089] Q rope =R n W Q E t ,

[0090] K rope =R n W K E t ,

[0091] V=W V E t

[0092] Subsequently, the attention score is calculated as follows:

[0093]

[0094] This integration ensures that the attention mechanism is guided by relative spatial relationships rather than fixed absolute positions. By combining rotational position encoding with dynamic embedding, the Rotational Attention Module (RAM) not only captures the relative spatial relationships between objects but also adapts to the dynamic changes of target states, thereby improving the overall performance of multi-object tracking.

[0095] S3. Perform spatial clustering on the tracking results through the spatial grouping attention module to obtain the tracking query results;

[0096] In a feasible implementation, spatial clustering is performed on the tracking results through a spatial grouping attention module to obtain tracking query results, including:

[0097] Compute the pairwise Euclidean distance matrix of the target bounding box to generate spatial proximity information;

[0098] Based on the distance matrix, the targets are grouped into spatially coherent groups through an iterative merging process;

[0099] Construct an attention mask to restrict the self-attention computation within the group and update the target embedding in combination with the rotation position encoding.

[0100] In one possible implementation, based on a distance matrix, the objects are grouped into spatially coherent groups through an iterative merging process, including:

[0101] Initialize each target as an independent group, and iteratively merge the groups based on the minimum average inter-group distance until the number of groups does not exceed the preset threshold;

[0102] The attention mask is used to restrict the self-attention mechanism within a group, allowing only targets within the same group to interact.

[0103] In one possible implementation, accurately tracking multiple targets in a dynamic environment is a challenging task due to the constant movement and interaction of the targets. Effective multi-target tracking requires powerful mechanisms to capture and exploit the spatial relationships between targets. The Spatial Grouped Attention Module (SGAM) is specifically designed to address these challenges, aiming to enhance the accuracy and efficiency of target state updates by modeling spatial relationships and improving tracking accuracy. Through the Spatial Grouped Attention Module (SGAM), a comprehensive solution is provided to maintain consistent and reliable tracking even in complex scenes.

[0104] The Spatial Group Attention Module (SGAM) achieves its goal through a structured approach that involves grouping tracking queries according to their bounding box locations and then updating their embeddings to reflect the interactions within the group. The target object of this invention is the tracking result TQ′ of the t-th frame t ={E′ t , B′ t}, where B′ t ={B1, B2, ..., B N} represents the result bounding box of the tth frame, where B i =(c x，i , c y，i , w i , h i ) represents the set B′ t A specific element in c corresponds to the i-th bounding box. x，i and c y，i Represents the x and y coordinates of the center of the bounding box.

[0105] Initially, the spatial grouped attention module (SGAM) computes the pairwise Euclidean distance matrix of all bounding boxes. The distance matrix D∈RN×N quantifies the spatial proximity between each pair of bounding boxes by:

[0106]

[0107] Using this distance matrix, the module organizes tracking queries into spatially coherent groups through an iterative merging process. Initially, each bounding box is assigned to its own group. The algorithm then iteratively merges the pair of groups with the smallest average inter-group distance until the number of groups does not exceed a predefined maximum threshold Gmax. This ensures that the resulting groups are spatially meaningful and facilitates local attention computation.

[0108] like Figure 2 As shown, an attention mask Γ∈{0,1}N×N is constructed to restrict the self-attention computation within these groups:

[0109]

[0110] where g i =g j Represents the bounding box B i and B j belong to the same group. This means they are spatially correlated and can interact with each other via the local attention mechanism within the group.

[0111] Generate positional embeddings using Rotational Positional Encoding (RoPE):

[0112] R=RoPE(B' t )

[0113] Where R represents the rotation position encoding corresponding to the generation. This embedding is combined with the output embedding E′t to form the query Q and key K of the self-attention mechanism:

[0114] Q=R·(W Q E′ t ),

[0115] K=R·(W K E′ t ),

[0116] V=W V E′ t

[0117] Then, we apply the self-attention (SA) mechanism using these queries, keys, and attention masks:

[0118] A=Self-Attention(Q,K,V,Γ)

[0119] Where A represents the attention output. The attention output A is integrated into the target embedding Et′ through residual connection and layer normalization (LN):

[0120] T′=LN(E′ t +(A))

[0121] Subsequently, two feed-forward networks (FFNs) with activation functions further optimize T′. Finally, the query feature Et′ is updated and assigned to the positive sample instance:

[0122] E t =LN(E′ t +(FFN(FFN(T′))))

[0123] B t =B′ t

[0124] The Spatial Group Attention Module (SGAM) effectively preserves local spatial relationships that are critical for accurate tracking by building a self-attention mechanism within groups that exhibit local spatial coherence. By leveraging position embeddings generated through RoPE (Rotational Position Embedding), which encode relative position information, the module is able to effectively handle overlapping and interacting targets, ensuring strong spatial perception and environmental adaptability. This design enables the Spatial Group Attention Module (SGAM) to achieve high-precision and highly reliable multi-target tracking performance, even in complex and dynamic real-world scenes.

[0125] S4. Using collective average loss CAL, we aggregate the prediction results of multiple frames in the video sequence and optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; based on the optimized modules, we perform relative position-aware Transformer multi-target tracking.

[0126] In a feasible implementation, the collective average loss CAL includes:

[0127] Match the predicted bounding box of each frame with the real annotation and calculate the classification loss, localization loss and intersection-over-union loss;

[0128] The multi-frame losses are normalized by the total number of targets to generate the total loss function and obtain the collective average loss CAL.

[0129] In one feasible implementation, the rotation attention module and the spatial grouping attention module are integrated into the Transformer decoder for end-to-end multi-object tracking.

[0130] In one possible implementation, training the proposed framework involves optimizing integrated modules - Rotational Attention Module (RAM) and Spatial Grouping Attention Module (SGAM) to efficiently perform multi-target tracking. The training process of the present invention aims to exploit the sequential nature of the tracking task and ensure that temporal information is accurately captured and utilized throughout the model. Following the MOTR\cite{MOTR} approach, the present invention adopts the collective average loss (CAL) to optimize the model of the present invention. Unlike traditional frame-by-frame loss calculation, the collective average loss (CAL) aggregates multiple predictions over the entire video sequence, thereby taking into account the temporal dependencies that are crucial for robust tracking. Specifically, for a video sequence consisting of T frames, the model generates a set of predictions Each of these Including predicted bounding boxes and the corresponding identity For real marking Indicates that the matching results are expressed as express.

[0131] The total loss for the entire video sequence is calculated by comparing the predictions with the ground truth and normalized by the total number of objects in all frames. It is formally expressed as:

[0132]

[0133] in Represents the total number of true targets in the nth frame. Here, and They represent the number of tracked targets and the number of newly detected targets in the nth frame, respectively.

[0134] The formula of the per-frame loss L is similar to the loss used in DETR, including classification, localization, and overlap metrics:

[0135]

[0136] in: represents the focal loss, which is used to handle the classification of object identity; represents the L1 loss, which is used to measure the difference between the predicted bounding box coordinates and the true bounding box coordinates; is the generalized intersection-over-union (GIoU) ​​loss, which is used to evaluate the relationship between the predicted bounding box and the true bounding box.

[0137] In a feasible implementation, Figure 2 RP 2 Trac framework (Relative Position Perception for Tracking) illustration. In frame 0, YOLOX detects the image to generate a detection query, while simultaneously extracting features through the backbone network and Transformer encoder. The detection query then interacts with image features in the Rotational Attention Module (RAM) to derive tracking results, and the tracking query is generated through the Spatial Grouped Attention Module (SGAM). In subsequent frames, the query consists of two parts: the detection query and the tracking query, and interacts with image features.

[0138] As shown in Table 1, Table 2 and Table 3 below: DanceTrack (dance tracking) data set: RP proposed in this invention 2 The Track framework demonstrates excellent performance on the DanceTrack dataset, achieving new state-of-the-art performance on all evaluation metrics, as shown in Table 1.

[0139] When training without the DanceTrack validation set, our method achieves a HOTA score of 71.4, an IDF1 score of 75.8, and an AssA score of 62.6. In this training configuration, these results surpass all existing methods.

[0140] Furthermore, using the DanceTrack validation set during training significantly improved its performance, achieving a HOTA score of 75.5, an IDF1 score of 81.3, and an AssA score of 67.9. Compared to the previous best-performing method, MOTRv2, these metrics improved by 2.1 points on HOTA, 5.3 points on IDF1, and 3.5 points on AssA. Furthermore, our method maintained high MOTA and DetA scores of 93.1 and 84.0, respectively, compared to MOTRv2's 92.1 and 83.7.

[0141] The consistent improvement across all metrics highlights the effectiveness of our framework in capturing and leveraging relative spatial relationships and dynamic object states. By integrating a rotational attention module (RAM) and a spatial grouping attention module (SGAM), we effectively address the unique challenges of multi-object tracking, improving tracking accuracy and robustness in dynamic and complex scenes.

[0142] SportMOT (Sports Multi-Object Tracking) dataset: On the large-scale SportsMOT dataset, the RP 2 The Track framework demonstrates excellent tracking performance, surpassing all existing state-of-the-art methods on multiple evaluation metrics. See Table 2 for details.

[0143] In a training setting using only the training set, our method achieves a HOTA score of 73.1, an IDF1 score of 76.0, and an AssA score of 63.2, outperforming previous methods such as DiffMOT, which achieved a HOTA score of 72.1, an IDF1 score of 72.8, and an AssA score of 60.5. This represents an improvement of 1.0 in HOTA, 3.2 in IDF1, and 2.7 in AssA.

[0144] When the SportsMOT validation set was included in the training process, its performance was further improved, achieving a HOTA score of 79.4, an IDF1 score of 84.9, and an AssA score of 72.2. Compared with DiffMOT (which had a HOTA score of 76.2, an IDF1 score of 76.1, and an AssA score of 65.1), our method improved HOTA by 3.2, IDF1 by 8.8, and AssA by 7.1.

[0145] Furthermore, our method achieves a competitive MOTA score of 93.0 and DetA score of 84.6 using only the training set, and 96.0 and 87.5, respectively, using both the training and validation sets (Train+Val). These results demonstrate the robustness and accuracy of our tracking system in a variety of dynamic scenarios.

[0146] Consistent improvements across all key metrics highlight our effectiveness in exploiting relative spatial relationships and dynamic target states, leading to improved multi-target tracking performance in complex sports environments.

[0147] MOT17 dataset: On the MOT17 dataset, the RP proposed in this paper 2Track demonstrates strong competitiveness among Transformer-based methods, as shown in Table 3. Our method achieves a HOTA score of 59.6, an AssA score of 61.7, and a DetA score of 57.8.

[0148] However, given the characteristics of the MOT17 dataset, such as its relatively small size and relatively simple pedestrian tracking features, our model still faces certain challenges on this dataset. Nevertheless, compared with existing Transformer-based methods, it improves on several key metrics, such as HOTA and IDF1.

[0149] Compared to MOTRv2, our method improves HOTA by 2.0 points and IDF1 by 3.7 points. However, compared to state-of-the-art convolutional neural network (CNN)-based methods, our Transformer-based method still lags behind in performance. This suggests that our method still has room for further improvement in the training setup of the MOT17 dataset.

[0150] Overall, while our Transformer-based approach has made progress on the MOT17 dataset, it still requires further optimization compared to traditional CNN-based methods, especially when dealing with complex scenarios that require strong detection performance. Our framework still has room for improvement in terms of accuracy and stability.

[0151] Table 1: Performance comparison with state-of-the-art methods on the DanceTrack test set. The best performance is marked in bold.

[0152]

[0153] Table 2: Performance comparison with state-of-the-art methods on the SportsMOT test set. The best performance is marked in bold.

[0154]

[0155] Table 3: Performance comparison with state-of-the-art methods on the MOT17 test set. The best performance is marked in bold.

[0156]

[0157] In this embodiment, a novel multi-target tracking framework, RP2Track, is proposed to enhance the model's ability to perceive spatial relationships between targets. RP2Track improves the model's ability to capture relative position information by introducing relative position encoding and adaptive perception modules. Its core components include a rotational attention module and a spatial grouping attention module.

[0158] The RAM module uses rotational position encoding to model relative positions. This rotational transformation encodes complex spatial relationships between objects, allowing the attention mechanism to focus on relative positions rather than fixed absolute positions, thereby improving the stability of target trajectories. The SGAM module uses a spatial grouping attention mechanism to cluster objects based on their relative positions, enabling local attention calculations and effectively improving tracking performance in dense scenes.

[0159] Figure 3 1 is a block diagram of a Transformer multi-target tracking device 300 based on relative position perception according to an exemplary embodiment. The device 300 is used for a Transformer multi-target tracking method based on relative position perception. Figure 3 The apparatus includes a feature extraction module 310, a rotation attention module 320, a spatial grouping attention module 330, and an optimization integration module 340.

[0160] The feature extraction module 310 is used to obtain the target to be detected, detect the target, and extract image features of the target;

[0161] The rotation attention module 320 is used to model the relative position relationship of the image features extracted from the target to obtain the tracking result;

[0162] The spatial grouping attention module 330 is used to perform spatial clustering on the tracking results to obtain the tracking query results.

[0163] The optimization integration module 340 is used to adopt the collective average loss CAL to aggregate the prediction results of multiple frames in the video sequence, optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; and perform relative position-aware Transformer multi-target tracking based on the optimized modules.

[0164] Optionally, a rotation attention module 320 is configured to scale the bounding box position information of the target by a preset factor to generate a scaled position vector;

[0165] Based on the diagonal rotation matrix of the rotation position encoding building block, the position embedding is rotated to generate relative position encoding;

[0166] Combining relative position encoding with target embedding, we compute attention scores guided by relative spatial relationships by modifying the query and key tensors.

[0167] Optionally, the rotation is based on the diagonal rotation matrix of the rotation position encoding building block, and the position embedding is rotated to generate a relative position encoding, including:

[0168] The position information of the markers is embedded in complex space and the relative position relationship is captured through rotation transformation;

[0169] Calculate the rotation angle based on the position embedding dimension and generate a block diagonal rotation submatrix;

[0170] The rotation sub-matrices are arranged along the block diagonal to form a block diagonal rotation matrix to generate relative position codes.

[0171] Optionally, a spatial grouping attention module 330 is used to calculate the pairwise Euclidean distance matrix of the target bounding box to generate spatial proximity information;

[0172] Based on the distance matrix, the targets are grouped into spatially coherent groups through an iterative merging process;

[0173] Construct an attention mask to restrict the self-attention computation within the group and update the target embedding in combination with the rotation position encoding.

[0174] Optionally, targets are grouped into spatially coherent groups through an iterative merging process based on a distance matrix, including:

[0175] Initialize each target as an independent group, and iteratively merge the groups based on the minimum average inter-group distance until the number of groups does not exceed the preset threshold;

[0176] The attention mask is used to restrict the self-attention mechanism within a group, allowing only targets within the same group to interact.

[0177] Optionally, a collective average loss CAL, comprising:

[0178] Match the predicted bounding box of each frame with the real annotation and calculate the classification loss, localization loss and intersection-over-union loss;

[0179] The multi-frame losses are normalized by the total number of targets to generate the total loss function and obtain the collective average loss CAL.

[0180] Optionally, the rotational attention module and the spatial grouping attention module are integrated into the Transformer decoder for end-to-end multi-object tracking.

[0181] In this paper, Rotational Position Embedding (RoPE) is introduced for the first time into the field of multi-object tracking (MOT), and a novel relative position encoding strategy specifically tailored for multi-object tracking scenarios is proposed. By incorporating a rotation-based position-aware mechanism, this approach significantly increases the model's sensitivity to the spatial relationships between objects, thereby improving tracking performance.

[0182] We propose a spatial grouping attention module that enhances focus on local features in crowded scenes by building a localized attention mechanism. This design effectively improves the model's ability to perceive occluded objects, demonstrating exceptional performance in complex interactive scenarios.

[0183] Figure 4 is a structural diagram of a Transformer multi-target tracking device based on relative position perception provided by an embodiment of the present invention, such as Figure 4 As shown, the Transformer multi-target tracking device based on relative position perception can include the above Figure 3 The Transformer multi-target tracking device based on relative position perception is shown. Optionally, the Transformer multi-target tracking device based on relative position perception 410 may include a first processor 2001.

[0184] Optionally, the Transformer multi-target tracking device 410 based on relative position perception may further include a memory 2002 and a transceiver 2003 .

[0185] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0186] The following combination Figure 4 The components of the Transformer multi-target tracking device 410 based on relative position perception are described in detail:

[0187] The first processor 2001 is the control center of the relative position sensing-based Transformer multi-target tracking device 410, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0188] Optionally, the first processor 2001 can perform various functions of the Transformer multi-target tracking device 410 based on relative position perception by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0189] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4CPU0 and CPU1 are shown in FIG.

[0190] In a specific implementation, as an embodiment, the Transformer multi-target tracking device 410 based on relative position perception may also include multiple processors, such as Figure 4 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0191] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0192] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the memory 2002 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0193] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0194] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0195] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0196] It should be noted that Figure 4 The structure of the Transformer multi-target tracking device 410 based on relative position perception shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0197] In addition, the technical effects of the Transformer multi-target tracking device 410 based on relative position perception can refer to the technical effects of the Transformer multi-target tracking method based on relative position perception described in the above method embodiment, and will not be repeated here.

[0198] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0199] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0200] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensor. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0201] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0202] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0203] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0204] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0205] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0206] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0207] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A Transformer multi-target tracking method based on relative position perception, characterized in that: The method comprises: S1. Obtain the target to be detected, detect the target, and extract image features of the target; S2, model the relative position relationship of the image features extracted from the target through the rotation attention module to obtain the tracking result; S3. Perform spatial clustering on the tracking results through the spatial grouping attention module to obtain the tracking query results; S4. Using collective average loss CAL, we aggregate the prediction results of multiple frames in the video sequence and optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; based on the optimized modules, we perform relative position-aware Transformer multi-target tracking.

2. The method according to claim 1, characterized in that In S2, the tracking results are obtained by rotating the attention module and modeling the relative position relationship of the image features extracted from the target, including: Scale the target's bounding box position information by a preset factor to generate a scaled position vector; Based on the diagonal rotation matrix of the rotation position encoding building block, the position embedding is rotated to generate relative position encoding; Combining relative position encoding with target embedding, we compute attention scores guided by relative spatial relationships by modifying the query and key tensors.

3. The method according to claim 2, characterized in that Based on the rotation position encoding building block, the diagonal rotation matrix is ​​used to rotate the position embedding to generate the relative position encoding, including: The position information of the markers is embedded in complex space and the relative position relationship is captured through rotation transformation; Calculate the rotation angle based on the position embedding dimension and generate a block diagonal rotation submatrix; The rotation sub-matrices are arranged along the block diagonal to form a block diagonal rotation matrix to generate relative position codes.

4. The method according to claim 2, characterized in that The tracking results are spatially clustered through the spatial grouping attention module to obtain tracking query results, including: Compute the pairwise Euclidean distance matrix of the target bounding box to generate spatial proximity information; Based on the distance matrix, the targets are grouped into spatially coherent groups through an iterative merging process; Construct an attention mask to restrict the self-attention computation within the group and update the target embedding in combination with the rotation position encoding.

5. The method according to claim 4, characterized in that Based on the distance matrix, the targets are grouped into spatially coherent groups through an iterative merging process, including: Initialize each target as an independent group, and iteratively merge the groups based on the minimum average inter-group distance until the number of groups does not exceed the preset threshold; The attention mask is used to restrict the self-attention mechanism within a group, allowing only targets within the same group to interact.

6. The method according to claim 1, characterized in that Collective average loss of CAL, including: Match the predicted bounding box of each frame with the real annotation and calculate the classification loss, localization loss and intersection-over-union loss; The multi-frame losses are normalized by the total number of targets to generate the total loss function and obtain the collective average loss CAL.

7. The method according to claim 1, characterized in that The rotation attention module and spatial grouping attention module are integrated into the Transformer decoder for end-to-end multi-object tracking.

8. A Transformer multi-target tracking device based on relative position perception, wherein the Transformer multi-target tracking device based on relative position perception is used to implement the Transformer multi-target tracking method based on relative position perception according to any one of claims 1 to 7, characterized in that: The device comprises: The feature extraction module is used to obtain the target to be detected, detect the target, and extract image features of the target; The rotation attention module is used to model the relative position relationship of the image features extracted from the target to obtain tracking results; The spatial grouping attention module is used to spatially cluster the tracking results to obtain the tracking query result optimization integration module, which is used to adopt the collective average loss CAL to aggregate the prediction results of multiple frames in the video sequence and optimize the temporal dependency of the rotation attention module and the spatial grouping attention module; based on the optimized module, Transformer multi-target tracking based on relative position perception is performed.

9. A Transformer multi-target tracking device based on relative position perception, comprising: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, any one of the relative position-aware Transformer multi-target tracking methods according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any one of the relative position perception-based Transformer multi-target tracking methods according to any one of claims 1 to 7.

Images