A relative position perception-based transformer multi-target tracking method

Abstract

The present application relates to the technical field of computer vision, and particularly relates to a relative position perception-based Transformer multi-target tracking method.The method comprises the following steps: modeling the relative position relationship of the image features of the target extraction through a rotation attention module to obtain a tracking result; performing spatial clustering on the tracking result through a spatial grouping attention module to obtain a tracking query result; adopting a collective average loss (CAL) to aggregate the prediction results of multiple frames in a video sequence, and optimizing the time dependence of the rotation attention module and the spatial grouping attention module; and performing relative position perception-based Transformer multi-target tracking based on the optimized module.The present application aims to enhance the spatial relationship perception ability of the model.RP2Track improves the relative position information capturing ability of the model by introducing a rotation position encoding and a spatial grouping attention module.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a relative position perception based Transformer multi-target tracking method. BACKGROUND

[0002] The research purpose of multi-target tracking algorithm in the field of computer vision is to develop a method that can automatically identify and track multiple targets in video or image. The main goals include the following aspects:

[0003] 1. Target detection and recognition: The primary task of multi-target tracking algorithm is to accurately detect and recognize multiple targets in the input image or video. This includes obtaining the position, bounding box, appearance features and category information of the target.

[0004] 2. Target tracking: The algorithm needs to track the movement of the target in consecutive image or video frames. This involves the consistency and continuity of the target in time, and the algorithm needs to predict and update the position of the target in the current frame based on the position information of the target in the previous frame.

[0005] 3. Robustness and real-time performance: Multi-target tracking algorithm needs to have robustness, which can cope with various complex situations in the scene, such as light changes, cluttered backgrounds, target deformation, etc. At the same time, the algorithm also needs to have real-time performance, which can efficiently track the target in real-time video stream.

[0006] The purpose of researching multi-target tracking algorithm is to improve the computer's understanding and analysis ability of multiple targets in video or image, and to provide more accurate and efficient target detection and tracking technology for various application fields such as video surveillance, autonomous driving, intelligent transportation, etc.

[0007] Most of the existing efficient multi-target tracking algorithms are divided into two modes, TBD (tracking by detection) mode and JDT (joint detection and tracking) mode. TBD mode adopts the way of detecting target first, then associating information in time dimension to realize target tracking. The mainstream TBD mode algorithms are SORT, DeepSORT, ByteTrack, etc., while JDT mode simultaneously performs target detection, target identity information extraction and association in one model. The mainstream JDT mode algorithms are JDE, TraDes, MOTR, etc.

[0008] The tracking-by-detection (TBD) paradigm treats multi-object tracking as two separate tasks: object detection and object tracking. First, a high-quality bounding box is generated using an effective detector, and then these boxes are processed by an association model that considers appearance and motion cues. The association process uses pairwise appearance re-identification (Re-ID) similarity for appearance matching and various heuristic methods, such as intersection over union (IoU) or Kalman filtering, for motion-based association. Although the TBD paradigm has been widely used for multi-object tracking, its two-stage framework hinders end-to-end optimization during training.

[0009] In contrast, the joint detection and tracking (JDT) paradigm aims to address this limitation by implementing both detection and association of multiple objects in a single stage. However, JDT-based methods, such as MOTR and TraDeS, face challenges in optimization as there is an inherent competition in appearance learning, with detection increasing inter-class variance and ReID increasing intra-class variance, leading to poor tracking performance.

[0010] In recent years, multi-object tracking (MOT) has made significant progress in the field of computer vision, with wide applications in video surveillance, autonomous driving, and robot navigation. The core goal of MOT is to accurately detect and track multiple objects in video sequences while maintaining the accuracy of detection and temporal consistency of object identities. However, MOT still faces many challenges due to factors such as occlusion, deformation, and lighting changes.

[0011] Deep learning-based methods have significantly improved the performance of MOT, especially with the introduction of the Transformer architecture. The self-attention mechanism of the Transformer can effectively capture global dependencies between different positions in a sequence, improving the accuracy of object association and significantly enhancing the performance of MOT in complex scenarios.

[0012] Currently, MOT methods using the Transformer architecture mainly draw on techniques from object detection, but directly applying object detection methods to MOT often overlooks the unique requirements of the tracking task. Object detection focuses on identifying individual objects in an image and annotating their bounding boxes, while MOT not only needs to identify objects but also ensure temporal consistency between frames and maintain robustness in the presence of multiple object interactions and dynamic changes.

[0013] However, existing Transformer methods often overlook the importance of relative position information in multi-object tracking. Relative position information, such as the relative distance, direction, and motion patterns between objects, is crucial for maintaining identity consistency and addressing occlusion issues. Many Transformer methods mainly rely on absolute position information, with insufficient modeling of relative position relationships, leading to limited performance in complex motion scenarios.

[0014] For example, in highly occluded scenes, traditional methods are prone to identity confusion or loss due to only capturing local features of the target, thereby affecting the stability and accuracy of tracking. Although the Transformer method improves the modeling ability of global dependence, it pays insufficient attention to local fine-grained information, making it difficult to effectively handle the occlusion problem.

[0015] Currently, the MOT framework based on Transform largely inherits the Transform model used for the target detection task. In these frameworks, each target is treated as an independent entity. For the t-th frame in a series of tracking videos, the model first inherits the detection query in the Transform framework used in target detection and additionally introduces a tracking query from the previous frame and a temporal aggregation network, enabling the model to learn target tracking in an adaptive end-to-end manner. Here, E represents the embedding of the target, and B represents the location information. However, when the target query interacts with the features, these methods still rely on absolute sine position encoding and standard attention mechanisms. This approach has certain limitations, especially in scenarios involving frequent interactions between targets. At the same time, the attention data caliber in Transform leads to the defect of local attention, making it difficult for current MOT methods based on Transform to accurately track targets during occlusion and interaction. In these cases, adjacent targets are often mistakenly identified as the same target or cannot be distinguished at all. SUMMARY

[0016] To solve the technical problem in the prior art that in highly occluded scenes, traditional methods are prone to identity confusion or loss due to only capturing local features of the target, thereby affecting the stability and accuracy of tracking. Although the Transformer method improves the modeling ability of global dependence, it pays insufficient attention to local fine-grained information, making it difficult to effectively handle the occlusion problem, the embodiments of the present application provide a Transformer multi-target tracking method and device based on relative position perception. The technical solution is as follows:

[0017] On the one hand, a Transformer multi-target tracking method based on relative position perception is provided, characterized in that the method comprises:

[0018] S1, acquiring a target to be detected, detecting the target, and extracting image features of the target;

[0019] S2, modeling the relative position relationship of the image features of the target extracted by the rotation attention module, and obtaining a tracking result;

[0020] S3, performing spatial clustering on the tracking result by a spatial grouping attention module to obtain a tracking query result;

[0021] S4, adopting a collective average loss CAL to aggregate prediction results of multiple frames in a video sequence, and optimizing time dependence of the rotation attention module and the spatial grouping attention module; and performing relative position perception based Transformer multi-target tracking based on the optimized modules.

[0022] Optionally, in S2, the tracking result is obtained by modeling the relative position relationship of the target extracted image features through the rotation attention module.

[0023] The bounding box position information of the target is scaled by a preset factor to generate a scaled position vector.

[0024] A diagonal rotation matrix based on a rotation position encoding building block is used to perform rotation transformation on the position embedding to generate relative position encoding.

[0025] The relative position encoding is combined with the target embedding to calculate attention scores guided by relative spatial relationship by modifying the query and key tensors.

[0026] Optionally, the relative position encoding is generated by rotating the position embedding based on a diagonal rotation matrix of a rotation position encoding building block, including:

[0027] The marked position information is complexly embedded in a space, and the relative position relationship is captured through rotation transformation.

[0028] The rotation angle is calculated according to the dimension of the position embedding to generate a block diagonal rotation sub-matrix.

[0029] The rotation sub-matrix is arranged along the block diagonal line to form a block diagonal rotation matrix to generate the relative position encoding.

[0030] Optionally, the tracking query result is obtained by performing spatial clustering on the tracking result through a spatial grouping attention module, including:

[0031] The pair-wise Euclidean distance matrix of the target bounding box is calculated to generate spatial proximity information.

[0032] Based on the distance matrix, the targets are grouped into spatially contiguous groups through an iterative merging process.

[0033] An attention mask is constructed to limit intra-group self-attention calculation, and the target embedding is updated in combination with the rotation position encoding.

[0034] Optionally, the targets are grouped into spatially contiguous groups through an iterative merging process based on the distance matrix, including:

[0035] Initialize each target as an independent group, iteratively merge groups based on the minimum average inter-group distance until the number of groups does not exceed a preset threshold;

[0036] Limit the intra-group self-attention mechanism by attention mask, allowing only targets within the same group to interact.

[0037] Optionally, the collective average loss CAL includes:

[0038] Match the predicted bounding box of each frame with the true label, calculate the classification loss, positioning loss and intersection over union loss;

[0039] Normalize the multi-frame loss by the total number of targets to generate a total loss function and obtain the collective average loss CAL.

[0040] Optionally, the rotation attention module and the spatial grouping attention module are integrated in the Transformer decoder for end-to-end multi-target tracking.

[0041] In another aspect, a relative position-aware Transformer multi-target tracking device is provided, which is applied to the relative position-aware Transformer multi-target tracking method, and the device includes:

[0042] The feature extraction module is configured to acquire a target to be detected, detect the target, and extract image features of the target;

[0043] The rotation attention module is configured to model the relative position relationship of the image features of the target to obtain a tracking result;

[0044] The spatial grouping attention module is configured to spatially cluster the tracking result to obtain a tracking query result

[0045] The optimization integration module is configured to use the collective average loss CAL to aggregate the prediction results of multiple frames in a video sequence, optimize the temporal dependence of the rotation attention module and the spatial grouping attention module, and perform relative position-aware Transformer multi-target tracking based on the optimized modules.

[0046] Optionally, the data set initialization module is configured to guide the red team model to generate strategic multi-round confrontation prompts in a multi-thought-guided manner combined with pre-attack thinking data sets;

[0047] The pre-attack thinking data set classifies attack strategies into four categories: intention reversal, problem decomposition, role playing and mixed mode, and requires the red team model to output a strategic thinking process before generating an attack question.

[0048] In another aspect, a relative position perception based Transformer multi-object tracking device is provided, comprising a processor, a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above relative position perception based Transformer multi-object tracking methods.

[0049] In another aspect, a computer readable storage medium is provided, the storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement any one of the above relative position perception based Transformer multi-object tracking methods.

[0050] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0051] In the embodiments of the present application, the rotation position embedding (RoPE) is first introduced into the multi-object tracking (MOT) field, and a new relative position encoding strategy specially customized for the multi-object tracking scene is proposed. By incorporating the rotation-based position perception mechanism, this method significantly improves the sensitivity of the model to the spatial relationship between targets, thereby improving the tracking performance. A spatial grouping attention module is proposed, which enhances the attention to local features in crowded scenes by constructing a localized attention mechanism. This design effectively improves the ability of the model to perceive occluded targets, and especially exhibits excellent performance in complex interactive scenes. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 A flowchart of a relative position perception based Transformer multi-object tracking method provided by an embodiment of the present application;

[0054] Figure 2 A detailed flowchart of a relative position perception based Transformer multi-object tracking method provided by an embodiment of the present application

[0055] Figure 3 A block diagram of a relative position perception based Transformer multi-object tracking device provided by an embodiment of the present application;

[0056] Figure 4 A structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the present application will be described below with reference to the drawings.

[0058] In the embodiments of the present application, the words such as "example", "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0059] In the embodiments of the present application, sometimes the subscript such as W1 may be written in the form of non-subscript such as W1, and when the difference is not emphasized, the meanings expressed are consistent.

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0061] The embodiments of the present application provide a relative position perception based Transformer multi-target tracking method, which can be implemented by a relative position perception based Transformer multi-target tracking device. The relative position perception based Transformer multi-target tracking device can be a terminal or a server. Figure 1 As shown in the flow chart of the relative position perception based Transformer multi-target tracking method, as shown in Figure 1 The relative position perception based Transformer multi-target tracking method provided by the present application can include the following steps:

[0062] S1, obtaining a target to be detected, detecting the target, and extracting image features of the target;

[0063] S2, modeling the relative position relationship of the image features of the target extracted by the rotation attention module (RAM), and obtaining a tracking result.

[0064] In a feasible implementation manner, in S2, the relative position relationship of the image features of the target extracted by the rotation attention module is modeled, and a tracking result is obtained, including:

[0065] scaling the boundary box position information of the target by a preset factor to generate a scaled position vector;

[0066] Rotary Position Embedding (RoPE) is a technique used in natural language processing (NLP) to improve the understanding of word order in sequences by using a rotation transformation to better capture relative position information.

[0067] RoPE combines relative position information with target embeddings to calculate attention scores guided by relative spatial relationships by modifying query and key tensors.

[0068] In one possible implementation, generating relative position encoding based on a block diagonal rotation matrix of rotary position embedding building blocks includes:

[0069] Complex space embedding of labeled position information is performed to capture relative position relationships through a rotation transformation.

[0070] The rotation angle is calculated based on the dimension of the position embedding to generate a block diagonal rotation sub-matrix.

[0071] The rotation sub-matrix is arranged along the block diagonal to form a block diagonal rotation matrix to generate relative position encoding.

[0072] In one possible implementation, in natural language processing (NLP), position encoding is crucial for models like Transformer to understand the order of words in a sequence. Traditional position encoding relies on absolute positions, which can be limiting when modeling relative relationships between tokens. To address this issue, Rotary Position Embedding (RoPE) is introduced, which uses a rotation transformation to better capture relative position information. The core idea is to embed the position information of tokens into a complex space. By applying a rotation operation, the relative position information between tokens remains unchanged, thereby improving the model's ability to understand the sequence order.

[0073] As shown in Figure 2 , the present application applies the core concept of RoPE to the multi-target tracking task, aiming to process tracking objects by combining their position information Bt = [x, y, w, h] with tracking embeddings Et.

[0074] For the t-th frame, to ensure compatibility with the sine function used in the self-attention mechanism, the present application first scales Bt = [x, y, w, h] by a factor of 2π to obtain the scaled position vector p:

[0075] p t = 2πB t

[0076] Subsequently, the present application introduces a temperature parameter τ to adjust the decay of position information across different dimensions. For each dimension k, this decay is defined as:

[0077]

[0078] where f denotes the number of position features.

[0079] The present invention encodes the relative direction between objects by transforming the position embedding with a block-diagonal rotation matrix Rn. The construction process is as follows:

[0080] First, the rotation angle for each dimension is calculated:

[0081]

[0082] where d denotes the dimension of the position embedding.

[0083] Next, based on the position embedding and the calculated angles, a 2x2 rotation sub-matrix Rn,k is constructed for each dimension:

[0084]

[0085] where Rn,k is the position embedding of the kth dimension for the nth sample.

[0086] These 2x2 rotation sub-matrices are placed along the block-diagonal of a larger rotation matrix Rn:

[0087]

[0088] This block-diagonal structure enables the rotation matrix Rn to effectively apply a rotation transformation to the position embedding, encoding the relative direction between objects. Hereafter, the present invention will use RoPE to represent the implementation process of the above rotation position encoding. In the multi-target tracking task, the ability to track objects based on their relative spatial relationships is crucial for obtaining accurate results. As shown in the figure, the present invention integrates RoPE into the self-attention mechanism in the multi-target tracking decoder and the Spatial Grounding Attention Module (SGAM). Specifically, the present invention modifies the query (Qrope) and key (Krope) tensors by applying the rotation matrix:

[0089] Q rope = R n · W Q E t ,

[0090] K rope = R n · W K E t ,

[0091] V = W V E t

[0092] Subsequently, the attention score is calculated as follows:

[0093]

[0094] This integration ensures that the attention mechanism is guided by relative spatial relationships rather than fixed absolute positions. By combining rotational position encoding with dynamic embeddings, the Rotational Attention Module (RAM) not only captures the relative spatial relationships between objects but also adapts to the dynamic changes in target states, thereby improving the overall performance of multi-object tracking.

[0095] S3, performing spatial clustering on the tracking results through a spatial grouping attention module to obtain tracking query results;

[0096] In a feasible implementation, the spatial clustering of the tracking results through the spatial grouping attention module to obtain the tracking query results includes:

[0097] Calculating the pairwise Euclidean distance matrix of the target bounding boxes to generate spatial proximity information;

[0098] Based on the distance matrix, grouping the targets into spatially contiguous groups through an iterative merging process;

[0099] Constructing an attention mask to limit intra-group self-attention computation and updating target embeddings in combination with rotational position encoding.

[0100] In a feasible implementation, based on the distance matrix, the targets are grouped into spatially contiguous groups through an iterative merging process, which includes:

[0101] Initializing each target as an independent group, iteratively merging groups based on the minimum average inter-group distance until the number of groups does not exceed a pre-set threshold;

[0102] Restricting the intra-group self-attention mechanism through an attention mask, allowing only targets within the same group to interact.

[0103] In a feasible implementation, accurately tracking multiple targets in a dynamic environment is a major challenge due to the constant movement and interaction of targets. Effective multi-object tracking requires powerful mechanisms to capture and utilize spatial relationships between targets. The Spatial Grouping Attention Module (SGAM) is specifically designed to address these challenges, aiming to enhance the accuracy and efficiency of target state updates by modeling spatial relationships and improving tracking precision. Through the Spatial Grouping Attention Module (SGAM), a comprehensive solution is provided to maintain consistent and reliable tracking even in complex scenarios.

[0104] The spatial grouping attention module (SGAM) achieves its goal through a structured approach that involves grouping the tracked queries according to their bounding box positions and then updating their embeddings to reflect the interactions within the groups. The target object of the invention is the tracking results TQ' of the t-th frame t = {E' t , B' t}, where B' t = {B1, B2,..., B N} denotes the result bounding boxes of the t-th frame, and B i = (c x，i , c y，i , w i , h i ) denotes a specific element in the set B' t , corresponding to the i-th bounding box. c x，i and c y，i denote the x and y coordinates of the center of the bounding box.

[0105] Initially, the spatial grouping attention module (SGAM) computes a pairwise Euclidean distance matrix for all bounding boxes. The distance matrix D ∈ RN×N quantifies the spatial proximity between each pair of bounding boxes by:

[0106]

[0107] Using this distance matrix, the module organizes the tracked queries into spatially coherent groups through an iterative merging process. Initially, each bounding box is assigned to its own group. Then, the algorithm iteratively merges the pair of groups with the smallest average inter-group distance until the number of groups does not exceed a predefined maximum threshold Gmax. This ensures that the resulting groups are spatially meaningful and conducive to local attention computation.

[0108] As shown in Equation 1, an attention mask Γ ∈ {0, 1}N×N is constructed to restrict the self-attention computation within these groups: Figure 2

[0109]

[0110] where g i = g j denotes that the bounding boxes B i and B j belong to the same group. This means that they are spatially relevant and can interact through a local attention mechanism within the group.

[0111] The position embeddings are generated using a rotated position encoding (RoPE):

[0112] R = RoPE(B' t )​

[0113] where R represents the generated rotation position encoding. This embedding is combined with the output embedding E't to form the query Q and key K of the self-attention mechanism:

[0114] Q = R · (W Q E' t ),

[0115] K = R · (W K E' t ),

[0116] V = W V E' t

[0117] Then, a self-attention (SA) mechanism is applied using these queries, keys, and attention mask:

[0118] A = Self-Attention(Q, K, V, Γ)

[0119] where A represents the attention output. The attention output A is integrated into the target embedding Et' through a residual connection and layer normalization (LN):

[0120] T' = LN(E' t + (A))

[0121] Subsequently, two feed-forward networks (FFN) with activation functions further optimize T'. Finally, the query feature Et' is updated and assigned to the positive sample instance:

[0122] E t = LN(E' t + (FFN(FFN(T')))

[0123] B t = B' t

[0124] The spatial group attention module (SGAM) effectively preserves local spatial relationships that are crucial for accurate tracking by constructing a self-attention mechanism within groups that exhibit local spatial coherence. By utilizing position embeddings generated through RoPE (Rotary Position Embedding), which encodes relative position information, this module is able to effectively handle overlapping and interacting targets, ensuring strong spatial perception capabilities and environmental adaptability. This design enables the spatial group attention module (SGAM) to achieve high precision and highly reliable multi-target tracking performance, even in complex and dynamic real-world scenarios.

[0125] S4, adopt collective average loss CAL, aggregate the prediction results of multiple frames in the video sequence, optimize the time dependence of the rotation attention module and the spatial group attention module, and perform relative position perception-based Transformer multi-target tracking based on the optimized module.

[0126] In an available implementation, the collective average loss CAL includes:

[0127] The predicted bounding box of each frame is matched with the real label, and a classification loss, a positioning loss and an intersection over union loss are calculated.

[0128] The multi-frame loss is normalized according to the total number of targets to generate a total loss function, and the collective average loss CAL is obtained.

[0129] In an available implementation, the rotation attention module and the spatial group attention module are integrated in the Transformer decoder for end-to-end multi-target tracking.

[0130] In an available implementation, training the proposed framework involves optimizing the integrated modules, the rotation attention module (RAM) and the spatial group attention module (SGAM), to effectively perform multi-target tracking. The training process of the present application aims to exploit the sequential nature of the tracking task to ensure accurate capture and utilization of temporal information throughout the model. Following the approach of MOTR \cite{MOTR}, the present application employs a collective average loss (CAL) to optimize the model of the present application. Unlike traditional frame-by-frame loss calculation, the collective average loss (CAL) aggregates multiple predictions over the entire video sequence, thus taking into account the temporal dependence that is crucial for robust tracking. Specifically, for a video sequence consisting of T frames, the model generates a set of predictions where each includes the predicted bounding box and the corresponding identity The real label is denoted by and the matching result is denoted by .

[0131] The total loss of the entire video sequence is calculated by comparing the prediction results with the real labels and normalized by the total number of objects in all frames. Its formal representation is:

[0132]

[0133] where denotes the total number of real targets in the nth frame. Here, and denote the number of tracked targets and the number of newly detected targets in the nth frame, respectively.

[0134] The formula of the loss L of each frame is similar to the loss used in DETR, including classification, positioning and overlap metrics:

[0135]

[0136] wherein: represents the focal loss, used to process the classification of object identity; represents the L1 loss, used to measure the difference between the predicted bounding box coordinates and the real bounding box coordinates; is the generalized intersection over union (GIoU) loss, used to evaluate the difference between the predicted bounding box and the real bounding box.

[0137] In a feasible implementation manner, as Figure 2 RP 2 Trac framework (Relative Position Perception for Tracking) is illustrated. In the 0th frame, YOLOX detects the image to generate a detection query, while extracting features through the backbone network and the Transformer encoder. Then, the detection query interacts with the image features in the rotation attention module (RAM), so as to obtain a tracking result, and the tracking query is generated through the spatial grouping attention module (SGAM). In the subsequent frames, the query is composed of two parts of the detection query and the tracking query, and interacts with the image features.

[0138] As shown in Tables 1, 2 and 3: DanceTrack (Dance Tracking) dataset: the RP 2 The Track framework exhibits excellent performance on the DanceTrack dataset, and achieves new optimal levels on all evaluation indicators, as shown in Table 1.

[0139] Without training on the DanceTrack validation set, the method of the present application achieves a HOTA score of 71.4, an IDF1 score of 75.8 and an AssA score of 62.6. Under this training configuration, these results exceed all existing methods.

[0140] In addition, by using the DanceTrack validation set in the training process, its performance is significantly improved, reaching a HOTA score of 75.5, an IDF1 score of 81.3 and an AssA score of 67.9. Compared with the previously best-performing method MOTRv2, these indicators are improved by 2.1 points in HOTA, 5.3 points in IDF1 and 3.5 points in AssA. In addition, compared with 92.1 and 83.7 of MOTRv2, the method of the present application maintains higher MOTA and DetA scores of 93.1 and 84.0, respectively.

[0141] The consistent improvements across all metrics highlight the effectiveness of the framework in capturing and utilizing relative spatial relationships and dynamic target states. By integrating the Rotated Attention Module (RAM) and the Spatial Grouping Attention Module (SGAM), the unique challenges in multiple object tracking are effectively addressed, leading to improved tracking accuracy and robustness in dynamic and complex scenarios.

[0142] SportMOT (Sports Multi-Object Tracking) dataset: On the large-scale SportsMOT dataset, the RP 2 The Track framework demonstrates superior tracking performance, outperforming all existing state-of-the-art methods across multiple evaluation metrics, as detailed in Table II.

[0143] Under the training setting using only the training set, the method of the present application achieves a HOTA score of 73.1, an IDF1 score of 76.0, and an AssA score of 63.2, which is superior to previous methods such as DiffMOT, which has a HOTA score of 72.1, an IDF1 score of 72.8, and an AssA score of 60.5. This means an improvement of 1.0 in HOTA, 3.2 in IDF1, and 2.7 in AssA.

[0144] When incorporating the SportsMOT validation set during training, its performance is further improved, achieving a HOTA score of 79.4, an IDF1 score of 84.9, and an AssA score of 72.2. Compared with DiffMOT (which has a HOTA score of 76.2, an IDF1 score of 76.1, and an AssA score of 65.1), the method of the present application improves by 3.2 in HOTA, 8.8 in IDF1, and 7.1 in AssA.

[0145] In addition, the method of the present application maintains a competitive MOTA score of 93.0 and a DetA score of 84.6 under the setting of using only the training set; and 96.0 and 87.5, respectively, under the setting of using the training set plus the validation set (Train+Val). These results highlight the robustness and accuracy of the tracking system of the present application in various dynamic scenarios.

[0146] The consistent improvements across all key metrics emphasize the effectiveness in utilizing relative spatial relationships and dynamic target states, thereby improving multiple object tracking performance in complex sports environments.

[0147] MOT17 dataset: On the MOT17 dataset, the RP 2Track shows strong competitiveness in Transformer-based methods, as shown in Table Three. The method of the present application achieves a HOTA score of 59.6, an AssA score of 61.7, and a DetA score of 57.8.

[0148] However, considering the characteristics of the MOT17 dataset, such as its relatively small size and relatively simple pedestrian tracking features, the model of the present application still faces certain challenges on this dataset. Nevertheless, compared with existing Transformer-based methods, there is an improvement in several key indicators, such as HOTA and IDF1.

[0149] Compared with MOTRv2, the method of the present application improves HOTA by 2.0 and IDF1 by 3.7. However, compared with the most advanced convolutional neural network (CNN) -based method, the Transformer-based method of the present application still lags behind in performance. This indicates that the method of the present application still has room for further improvement in the training settings of the MOT17 dataset.

[0150] Overall, although the Transformer-based method of the present application has made progress on the MOT17 dataset, it still needs to be further optimized compared with traditional CNN-based methods, especially when dealing with complex scenarios that require strong detection performance. The framework of the present application still has room for improvement in terms of accuracy and stability.

[0151] Table One: Performance comparison with advanced methods on the DanceTrack test set. The best performance is highlighted in bold

[0152]

[0153] Table Two: Performance comparison with advanced methods on the SportsMOT test set. The best performance is highlighted in bold

[0154]

[0155] Table Three: Performance comparison with advanced methods on the MOT17 test set. The best performance is highlighted in bold

[0156]

[0157] In an embodiment of the present application, a novel multi-target tracking framework RP2Track is proposed, aiming to enhance the model's ability to perceive spatial relationships between targets. RP2Track improves the model's ability to capture relative position information by introducing relative position encoding and an adaptive perception module. Its core components include a rotation attention module and a spatial grouping attention module.

[0158] The RAM module models relative positions by using rotation position encoding, encodes complex spatial relationships between targets by rotation transformation, and makes the attention mechanism focus on relative positions rather than fixed absolute positions, thereby improving the stability of target trajectories. The SGAM module clusters targets according to the relative positions between the targets by using the spatial grouping attention mechanism, thereby realizing local attention calculation and effectively improving tracking performance in dense scenes.

[0159] Figure 3 is a relative position perception-based Transformer multi-target tracking device 300 block diagram shown according to an exemplary embodiment, the device 300 is used for a relative position perception-based Transformer multi-target tracking method. Referring to Figure 3 , the device includes a feature extraction module 310, a rotation attention module 320, a spatial grouping attention module 330, and an optimization integration module 340. Among them:

[0160] The feature extraction module 310 is configured to obtain a target to be detected, detect the target, and extract image features of the target;

[0161] The rotation attention module 320 is configured to model the relative position relationship of the image features of the target, and obtain a tracking result;

[0162] The spatial grouping attention module 330 is configured to spatially cluster the tracking result to obtain a tracking query result

[0163] The optimization integration module 340 is configured to use a collective average loss CAL to aggregate prediction results of multiple frames in a video sequence, and optimize the time dependence of the rotation attention module and the spatial grouping attention module; based on the optimized module, the relative position perception-based Transformer multi-target tracking is performed.

[0164] Optionally, the rotation attention module 320 is configured to scale the bounding box position information of the target by a preset factor to generate a scaled position vector;

[0165] A diagonal rotation matrix based on a rotation position encoding building block is used to rotate transform the position embedding to generate relative position encoding;

[0166] The relative position encoding is combined with the target embedding to calculate attention scores guided by relative spatial relationships by modifying the query and key tensors.

[0167] Optionally, the relative position encoding is generated by rotating the position embedding based on a diagonal rotation matrix of a rotation position encoding building block, including:

[0168] The marked position information is embedded in a complex space, and the relative position relationship is captured by rotation transformation;

[0169] The rotation angle is calculated according to the position embedding dimension, and a block diagonal rotation submatrix is generated;

[0170] The rotation submatrix is arranged along the block diagonal line to form a block diagonal rotation matrix, and a relative position code is generated.

[0171] Optionally, the spatial grouping attention module 330 is used to calculate the pair-wise Euclidean distance matrix of the target bounding box, and spatial proximity information is generated;

[0172] Based on the distance matrix, the target is grouped into spatially coherent groups through an iterative merging process;

[0173] An attention mask is constructed to limit intra-group self-attention calculation, and the target embedding is updated in combination with the rotated position code.

[0174] Optionally, based on the distance matrix, the target is grouped into spatially coherent groups through an iterative merging process, including:

[0175] Each target is initialized as an independent group, and the groups are iteratively merged based on the minimum average inter-group distance until the number of groups does not exceed a preset threshold;

[0176] The intra-group self-attention mechanism is limited by the attention mask, allowing only targets within the same group to interact.

[0177] Optionally, the collective average loss CAL includes:

[0178] The predicted bounding box of each frame is matched with the true label, and the classification loss, positioning loss and intersection over union loss are calculated;

[0179] The multi-frame loss is normalized according to the total number of targets to generate a total loss function, and the collective average loss CAL is obtained.

[0180] Optionally, the rotation attention module and the spatial grouping attention module are integrated into a Transformer decoder for end-to-end multi-target tracking.

[0181] In the embodiments of the application, the rotation position embedding (RoPE) is first introduced into the field of multi-target tracking (MOT), and a new relative position coding strategy specially tailored for the multi-target tracking scene is proposed. By incorporating a rotation-based position perception mechanism, this method significantly improves the model's sensitivity to the spatial relationships between targets, thereby improving tracking performance.

[0182] A spatial grouping attention module is proposed, which enhances the attention to local features in crowded scenes by constructing a localized attention mechanism. This design effectively improves the model's ability to perceive occluded targets, especially in complex interaction scenarios, and exhibits excellent performance.

[0183] Figure 4 is a structural schematic diagram of a relative position perception based Transformer multi-target tracking device provided by an embodiment of the present application, as shown in Figure 4 The relative position perception based Transformer multi-target tracking device can include the relative position perception based Transformer multi-target tracking apparatus shown in Figure 3 Optionally, the relative position perception based Transformer multi-target tracking device 410 can include a first processor 2001.

[0184] Optionally, the relative position perception based Transformer multi-target tracking device 410 can further include a memory 2002 and a transceiver 2003.

[0185] The first processor 2001 is connected with the memory 2002 and the transceiver 2003 through a communication bus.

[0186] The various constituent components of the relative position perception based Transformer multi-target tracking device 410 will be specifically introduced below: Figure 4

[0187] The first processor 2001 is the control center of the relative position perception based Transformer multi-target tracking device 410, and can be one processor or a collective term of multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), and can also be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0188] Optionally, the first processor 2001 can execute various functions of the relative position perception based Transformer multi-target tracking device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0189] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as Figure 4 ​CPU0 and CPU1 shown in FIG. 1.

[0190] In a specific implementation, as an embodiment, the relative position awareness based Transformer multi-target tracking device 410 can also include a plurality of processors, for example Figure 4 The first processor 2001 and the second processor 2004 shown in FIG. 2. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0191] The memory 2002 is configured to store a software program for implementing the scheme of the present application, and the first processor 2001 is configured to control the execution. For specific implementation, refer to the above-mentioned method embodiments, which will not be repeated here.

[0192] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through the interface circuit (not shown in FIG. 2) of the relative position awareness based Transformer multi-target tracking device 410, and the embodiments of the present application are not limited in this regard. Figure 4

[0193] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0194] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown separately in FIG. 2). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Figure 4

[0195] ​​Optionally, the transceiver 2003 can be integrated with the first processor 2001, or exist independently, and is coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the relative position perception based Transformer multi-target tracking device 410. Figure 4 The present embodiment is not limited in this regard.

[0196] It should be noted that, Figure 4 The structure of the relative position perception based Transformer multi-target tracking device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0197] In addition, the technical effects of the relative position perception based Transformer multi-target tracking device 410 can refer to the technical effects of the relative position perception based Transformer multi-target tracking method described in the above method embodiments, which will not be repeated here.

[0198] It should be understood that the first processor 2001 in the present embodiment can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0199] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0200] The above-described embodiments can be implemented in part or in whole through software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When loaded and executed by a computer, the computer instructions or computer programs can produce the processes or functions described above in accordance with the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, such as from a website, a computer, a server, or a data center to another website, computer, server, or data center through a wired (e.g., infrared, wireless, microwave, etc.) or wireless medium. The computer-readable storage medium can be any available medium or a collection of medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more of the available medium. The available medium can be a magnetic medium (e.g., a floppy diskette, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard drive.

[0201] It should be understood that the term "and / or" in this document is merely used to describe associated objects, and can represent three conditions: A and / or B, which can mean that only A exists, both A and B exist, or only B exists, where A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects, but can also represent an "and / or" relationship. The specific meaning can be understood according to the context before and after.

[0202] It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0203] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0204] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0205] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0206] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0207] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A relative position perception-based Transformer multi-target tracking method, characterized in that, The method comprises: S1, acquiring a target to be detected, detecting the target, and extracting image features of the target; S2, modeling the relative position relationship of the image features of the target extracted by the rotation attention module to obtain a tracking result; specifically comprising: scaling the position information of the bounding box of the target by a preset factor to generate a scaled position vector; constructing a block diagonal rotation matrix based on the relative position encoding, rotating the position embedding to generate a relative position encoding; combining the relative position encoding with the target embedding, modifying the query and key tensors to calculate the attention score guided by the relative spatial relationship; S3, spatial clustering of the tracking result by the spatial grouping attention module to obtain a tracking query result; specifically comprising: calculating the pairwise Euclidean distance matrix of the target bounding box to generate spatial proximity information; grouping the targets into spatially contiguous groups through an iterative merging process based on the distance matrix; constructing an attention mask to limit intra-group self-attention calculation, and updating the target embedding in combination with the relative position encoding; S4, using the collective average loss CAL to aggregate the prediction results of multiple frames in a video sequence to optimize the temporal dependence of the rotation attention module and the spatial grouping attention module; based on the optimized module, the relative position perception-based Transformer multi-target tracking is performed; wherein the rotation attention module and the spatial grouping attention module are integrated in the Transformer decoder for end-to-end multi-target tracking.

2. The method of claim 1, wherein, The diagonal rotation matrix based on the relative position encoding construction block is used to rotate the position embedding to generate a relative position encoding, which includes: performing complex space embedding on the marked position information to capture the relative position relationship through rotation transformation; calculating the rotation angle based on the position embedding dimension to generate a block diagonal rotation sub-matrix; arranging the rotation sub-matrix along the block diagonal to form a block diagonal rotation matrix to generate a relative position encoding.

3. The method of claim 1, wherein, Based on the distance matrix, the targets are grouped into spatially contiguous groups through an iterative merging process, which includes: initializing each target as an independent group, iteratively merging groups based on the minimum average inter-group distance until the number of groups does not exceed a preset threshold; limiting the intra-group self-attention mechanism through the attention mask, allowing only targets within the same group to interact.

4. The method of claim 1, wherein, The collective average loss CAL includes: matching the predicted bounding box of each frame with the true label to calculate the classification loss, positioning loss and intersection over union loss; normalizing the multi-frame loss by the total number of targets to generate a total loss function to obtain the collective average loss CAL.

5. A relative position-aware Transformer-based multi-object tracking device for implementing the relative position-aware Transformer-based multi-object tracking method according to any one of claims 1-4, characterized in that, The device comprises: a feature extraction module for acquiring a target to be detected, detecting the target, and extracting image features of the target; a rotation attention module for modeling the relative position relationship of the image features of the target extracted by the rotation attention module to obtain a tracking result; The spatial grouping attention module is used for spatial clustering of the tracking result, and a tracking query result optimization integration module is used for adopting a collective average loss (CAL) to aggregate prediction results of multiple frames in a video sequence, and time dependence of the rotation attention module and the spatial grouping attention module is optimized; and the optimized module is used for relative position perception based Transformer multi-target tracking. 6.A relative position perception based Transformer multi-target tracking device, comprising: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, implement any one of the relative position perception based Transformer multi-target tracking methods according to any one of claims 1-4. 7.A computer readable storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement any one of the relative position perception based Transformer multi-target tracking methods according to any one of claims 1-4.

Images