A long-term target tracking method based on timing query propagation

Abstract

A long-term target tracking method based on timing query propagation comprises the following steps: step one: obtain video sequence frames including a plurality of template frame sets and a plurality of search frame sets, and obtain template patch embeddings and search patch embeddings after flattening; step two: add position embeddings; step three: design target query vectors and background query vectors, input the first frame template patch embeddings and search patch embeddings obtained in step two and the designed target query vectors and background query vectors into a plurality of visual Transformer layers to perform attention calculation; step four: input the updated target query vectors and background query vectors in step three and the template patch embeddings and search patch embeddings of subsequent frames into the plurality of visual Transformer layers to perform attention calculation again; and step five: input the finally generated search frame patch embeddings into a prediction head to obtain target position coordinates. The present application can improve the robustness and accuracy of long-term tracking.

Description

Technical Field

[0001] This invention relates to the fields of pattern recognition and computer vision, and specifically to a long-term target tracking method based on temporal query propagation. Background Technology

[0002] Object tracking is a fundamental and crucial task in computer vision. Its goal is to predict the state of a target in subsequent frames, typically including its position and size, given the location or region of a target in the initial frame of a video sequence. Due to its core role, object tracking technology has been widely applied in several key areas, including video surveillance, human-computer interaction, autonomous driving, and robot navigation.

[0003] In recent years, thanks to the rapid development of deep learning technology, deep learning-based object tracking methods have achieved significant performance improvements and become the mainstream research approach. However, despite these great advances, existing deep learning object tracking methods still face many challenges in practical applications, especially when dealing with long-term tracking tasks, which limit their robustness and reliability. These challenges are mainly reflected in the following aspects:

[0004] Target appearance changes: During movement, the target will inevitably undergo appearance changes such as rotation, scaling, deformation, lighting changes, and even changes in its own posture. These changes may cause the target features extracted by the tracker to differ significantly from the initial template, thus causing tracking failure.

[0005] Background interference: In complex and dynamic background environments, there are often objects that look similar to the target, or there are interference factors such as sudden changes in lighting or scene switching. These can easily cause the tracker to drift and misjudge the background as the target.

[0006] Occlusion and target disappearance: During movement, the target may be completely or partially occluded by other objects, or even temporarily leave the field of view. Traditional trackers often struggle to resume tracking after the target reappears due to a lack of effective utilization of historical target information.

[0007] Model drift in long-term tracking: Some trackers based on online learning strategies, although they have a certain degree of adaptability, will update the model by taking the incorrect tracking results as positive samples if they encounter inaccurate predictions during long-term operation. Over time, this can easily accumulate errors, leading to model drift and ultimately reducing tracking performance.

[0008] As shown in Figure 1(a), most mainstream deep learning target tracking methods are currently trained based on single-frame image pairs. This means they use a template image containing the target and a search image of the target to learn the target's feature representation. This training paradigm simplifies the target tracking task into a similarity matching problem between image pairs, with the core being learning the visual similarity between candidate regions in the template image and the search image. However, this training method based on single-frame image pairs essentially treats video frames as independent static images, ignoring the inherent temporal dimension of the video sequence and the rich temporal information between frames. This neglect of temporal information makes it difficult for the tracker to effectively model the dynamic changes of the target during motion, thus proving inadequate in dealing with challenges such as changes in target appearance and background interference.

[0009] While some target tracking methods attempt to incorporate temporal information, their primary approach is to dynamically update the template image, replacing the initial template image with image patches cropped from predictions in subsequent tracking frames. The initial intention of this dynamic update strategy is to enable the model to learn target features more similar to the current frame. However, this strategy has inherent drawbacks: Firstly, when tracking predictions are unreliable, such as during drift or target occlusion, the updated template image may no longer contain the target being tracked, instead introducing background noise and accelerating tracking failure. Secondly, methods based on dynamic template updates often require careful design and adjustment of complex hyperparameters, such as the template update interval, update confidence threshold, and fusion weights. These hyperparameter settings significantly impact tracking performance and lack universality, requiring adjustments for different scenarios, increasing algorithm complexity and deployment difficulty.

[0010] Furthermore, in real-world tracking scenarios, the changing trends of the target region and the background region often differ significantly. For example, the target may undergo rapid deformation or movement, while the background may remain relatively stable, or there may be dynamic disturbances in the background unrelated to the target. However, most existing tracking algorithms often employ the same learning strategy when learning target and background features, failing to consider the differences in their changing characteristics. This results in an ineffective decoupling of target and background feature learning, thus limiting further improvements in tracking performance. This is akin to using the exact same standard to measure two things when learning to distinguish them, ignoring their individual characteristics.

[0011] The consecutive frames in a video sequence contain rich temporal information, including motion patterns and appearance evolution. Especially when dealing with challenges such as drastic changes in target appearance and complex background interference, temporal information can effectively supplement appearance representation learning, providing additional discriminative cues and significantly improving the robustness and accuracy of long-term target tracking. Existing techniques either ignore temporal information or utilize it in complex and error-prone ways (such as dynamic template updates), and lack effective modeling of the differences between target and background changes. Therefore, there is an urgent need for a target tracking method that can effectively and concisely utilize the temporal information of video sequences, avoid introducing complex hyperparameters, and distinguish between the learning target and background change characteristics, in order to overcome the shortcomings of existing techniques and achieve more robust and accurate long-term target tracking. Summary of the Invention

[0012] To address the shortcomings of existing target tracking methods in the background art, such as ignoring temporal information of video sequences, the deficiencies of dynamic template update strategies, and the failure to effectively decouple target and background learning, the present invention aims to propose a novel long-term target tracking method based on temporal query propagation, which aims to:

[0013] 1) Addressing the issue of insufficient utilization of temporal information in existing methods for video sequences. A novel temporally dense video frame sampling method is proposed for training target tracking algorithms, expanding the model's input from traditional single-frame image pairs to continuous, temporally correlated video sequences. This aims to enable the model to effectively learn and model the contextual information and cross-frame correlations between consecutive video frames, overcoming the inherent limitations of single-frame image pair training in temporal modeling.

[0014] 2) Constructing a more robust temporal target representation to achieve decoupled learning of target and background features: A target tracking algorithm based on two types of propagable query vectors is proposed: a target query vector for accurately representing the dynamic changes of the target's salient region over time, and a background query vector for finely representing the dynamic changes of the background region over time. By explicitly introducing and propagating these two types of query vectors in the video sequence, the target tracking task is reformulated as a sequence propagation task of these two types of query vectors. This allows for the accurate capture of the target's dynamic evolution in the temporal domain and trajectory changes in the spatial domain through online propagation of query vectors, improving the model's robustness to changes in target appearance and background interference.

[0015] By introducing target query vectors and background query vectors separately and performing time-series propagation independently, the aim is to decouple the learning of the changing characteristics of the target and the background, overcome the suboptimal situation caused by the indiscriminate learning of the two in existing technologies, improve the model's ability to distinguish between the target and the background, and reduce tracking drift caused by background interference.

[0016] 3) Avoid introducing complex hyperparameters and unstable dynamic template update strategies: The time-series query propagation method proposed in this invention does not require the design and adjustment of complex hyperparameters, nor does it rely on dynamic template update strategies that may introduce errors, thus providing a more stable, reliable and easy-to-deploy target tracking solution.

[0017] In summary, this invention aims to provide a target tracking method that can effectively utilize the temporal information of video sequences to improve the robustness and accuracy of long-term tracking, and overcome the shortcomings of existing technologies, thus providing new ideas for the development of target tracking technology in the field of computer vision.

[0018] This invention proposes a long-term target tracking method based on temporal query propagation. Its core lies in utilizing target query vectors and background query vectors to propagate between consecutive video frames, implicitly learning and modeling the temporal dynamic changes of the target and background. The specific steps of the method are as follows:

[0019] Step 1: Video sequence segmentation and patch embedding: Obtain a set of template frames including multiple frames. and multi-frame search frame set The video sequence is divided into frames, each frame is flattened into a series of image patches, and then each image patch is flattened into a one-dimensional vector. A trainable linear projection network is then used. The flattened template frame image patch and the search frame image patch are mapped to a high-dimensional latent space to obtain the template patch embedding. and search patch embedding ;

[0020] Step 2: Add Position Embedding: Embed the template patch obtained in Step 1 and search patch embedding Add learnable position embeddings respectively and ;

[0021] Step 3: Initial Query and Transformer Interaction: Designing the Target Query Vector Background query vector The target query vector aims to learn and represent the salient features of the target object, while the background query vector aims to learn and represent the features of the background region. The first frame template patch obtained in step two is then embedded. Search patch embedding With the target query vector of the design Background query vector Multiple visual Transformer layers are input together for attention calculation;

[0022] Step 4: Time-series query propagation and Transformer iteration: Update the target query vector in Step 3. Background query vector Embedded template patch with subsequent frames and search patch embedding The input is fed into multiple visual Transformer layers, and attention is calculated again; the target query vector is repeated. Background query vector Update and input the visual Transformer layer, that is, continuously use the Transformer layer to perform attention calculations between consecutive video frames;

[0023] Step 5: Embed the search frame patch generated in Step 4 after multiple Transformer iterations. Input the prediction header to obtain the target location coordinates.

[0024] The beneficial effects of the technical solution of this invention:

[0025] 1) This invention proposes a novel temporally dense video sequence sampling method for target tracking algorithm training, which expands the model input from single-frame image pairs to continuous video sequences, thereby modeling the contextual information and cross-frame correlation of continuous video frames.

[0026] This invention innovatively proposes a temporally dense video sequence sampling method for training target tracking algorithms, changing the previous method's reliance on single-frame image pairs for training. As shown in Figure 1(a), previous target tracking algorithms used temporally sparse sampling for training, i.e., sampling only isolated single-frame image pairs and simplifying the target tracking task to visual similarity matching between template and search image pairs. While this method can learn some visual representation capabilities, its inherent temporal sparsity determines its fundamental deficiency in utilizing the rich temporal information in video sequences, causing the model to only focus on the appearance similarity of the target and unable to fully explore and utilize the correlation between frames. In contrast, the temporally dense video frame sampling method proposed in this invention, as shown in Figure 1(b), samples consecutive template and search frame images as model input, allowing the model to access richer temporal contextual information. Under this new input paradigm, the model can effectively learn and model the rich temporal information such as motion patterns and appearance evolution contained in the preceding and following frames of the video sequence. This overcomes the problem of poor long-term tracking robustness caused by insufficient utilization of temporal information in traditional methods, and significantly improves the stability and accuracy of target tracking.

[0027] 2) This invention proposes a target tracking algorithm based on two types of query vectors for temporal propagation: a target query vector characterizing the temporal changes of the target's salient region and a background query vector characterizing the temporal changes of the background region. Target tracking is reformulated as a sequence propagation task of these two types of query vectors, capturing the spatiotemporal trajectory relationship of the target through online propagation of query vectors. This effectively decouples the temporal information learning of the target and the background, improving the ability to resist background interference.

[0028] This invention innovatively proposes a target tracking algorithm based on temporal propagation of two types of query vectors: a target query vector representing the temporal changes of the salient region of the target, and a background query vector representing the temporal changes of the background region. Unlike previous methods that treated target tracking as a simple visual similarity matching task, this invention reformulates target tracking as a propagation task of these two types of query vectors within a video frame sequence, capturing the spatiotemporal trajectory relationship of the target through the online propagation of query vectors. Specifically, as shown in Figure 1(b), based on continuous video frame input, this invention models target tracking as an autoregressive propagation process of query vectors within video frames. The video frame input provides the foundation for visual representation learning, while the propagation of query vectors focuses on the learning of temporal representations. The synergistic effect of these two methods achieves superior tracking performance. More importantly, in real-world tracking scenarios, the changing trends of the target region and the background region often differ significantly; that is, the rates and patterns of change between them are usually different. However, existing tracking algorithms often treat the learning of the target and background indiscriminately, resulting in an ineffective decoupling of feature learning and creating a bottleneck for performance optimization. To address this issue, this invention further designs the query vectors into two categories: target query vectors and background query vectors, and enables these two types of query vectors to undergo temporal autoregressive learning independently. This design cleverly achieves decoupled learning of target and background temporal information, avoiding mutual interference during their feature learning processes. This significantly improves the model's ability to distinguish between the target and the background, effectively suppresses background interference, and reduces tracking drift. Attached Figure Description

[0029] Figure 1(a) is a schematic flowchart of a target tracking method in the prior art that uses only a single frame image for training;

[0030] Figure 1(b) is a schematic flowchart of the long-term target tracking method based on temporal query propagation, which is trained using multi-frame video sequences, proposed in this invention.

[0031] Figure 2 A schematic flowchart illustrating attention computation for time-series query updates introduced according to the present invention is shown. Detailed Implementation

[0032] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0033] To fully understand the temporally dense video sequence sampling method proposed in this invention, it is necessary to first review the temporally sparse image pair sampling method.

[0034] As shown in Figure 1(a), the traditional temporal sparse image pair sampling method independently samples a template frame image from the video sequence. and a search frame image , in , These are the height and width of the template frame image, respectively. , The search involves determining the height and width of the image frame. Then, this pair of images... and Input target tracking algorithm Processing is performed to predict the bounding box coordinates of the target in the current search frame. ,Right now:

[0035] ,

[0036] if For a twin tracking network, it needs to go through three stages: visual feature extraction, feature interaction, and bounding box prediction. However, if... For a single-stream tracking network, it consists of only a visual Transformer backbone network and a prediction head network, where the backbone network performs both visual feature extraction and interaction steps.

[0037] against In the case of a single-stream tracking network, the visual Transformer backbone network receives a series of image patch embeddings as input. Specifically, it uses reference frame images... and search frame images Divided into The image patch is then passed through a trainable linear projection network. To generate template image tag sequences and search image tag sequences .in It is a labeled dimension. , Then these image tags are concatenated and input into... In the visual Transformer layer, feature extraction and interaction are performed simultaneously. Each visual Transformer layer... This includes a multi-head attention mechanism and a multilayer perceptron. The first... The forward propagation process of the layered visual Transformer layer can be described as follows:

[0038] ,

[0039] in, Indicates by the first The template frame-search frame image pair label sequence generated by the layer vision Transformer layer.

[0040] While the aforementioned temporally sparse image pair sampling method can construct a relatively simple target tracking framework, its fundamental flaw lies in the fact that the tracking model only focuses on the visual similarity of the target within a single frame, lacking the ability to establish temporal correlations between consecutive frames. This neglect of temporal information significantly hinders the robustness of target tracking in long video sequences. Because the model cannot effectively utilize the target's temporal trajectory and appearance changes, it is prone to tracking failure or drift when faced with complex situations such as target occlusion, rapid movement, and appearance deformation.

[0041] Temporally dense video sequence sampling

[0042] Unlike traditional temporally sparse image pair sampling methods, this invention expands the input of the target tracking algorithm from temporally isolated image pairs to temporally continuous video sequences, thereby enabling direct modeling of temporal information. Furthermore, this invention innovatively introduces two specially designed query vectors: target query vectors... and background query vector Based on the time-dense video sequence input and dual-query vector mechanism, the tracking process of this invention can be expressed as follows:

[0043] ,

[0044] in, Indicates length is template frame sequence, Indicates length is The search frame sequence, The coordinates of the predicted bounding box in the current search frame. It is a single-stream tracking network.

[0045] When sampling time-dense video sequences, instead of simply selecting adjacent frames, a larger sampling interval than traditional methods is used. Within this larger sampling interval, multiple video frames are randomly sampled to construct a template frame sequence of arbitrary length. and search frame sequence The core reason for using a larger sampling interval is to enable the model to learn the temporal relationships inherent in video sequences over a longer time scale. By sampling video frames with temporal relationships over a longer time span, the tracking model can effectively learn stronger and more robust temporal correlation information, thereby better coping with challenges such as changes in target appearance and occlusion.

[0046] Target query and background query design

[0047] As analyzed in the background section, in actual tracking scenarios, the changing trends of the target region and the background region often differ significantly. The target may undergo rapid deformation or movement, while the background is relatively stable or subject to dynamic interference unrelated to the target. To address the problem of indiscriminate learning of the target and background in existing technologies and to achieve decoupled learning of the temporal information of the target and background, this invention innovatively designs a target query vector. and background query vector These are used to model the temporal dynamic changes of the target and the background, respectively.

[0048] Target query vector The target query vector is randomly initialized as a one-dimensional vector. During subsequent tracking, this vector propagates through a continuous video sequence and interacts with the image features of each frame. Through this propagation and interaction mechanism, the target query vector can progressively aggregate and compress key trajectory information and appearance change information of the target in each frame, ultimately forming an effective representation of the target's temporal dynamic changes.

[0049] Unlike the target query vector, the background query vector The design fully considers the characteristics of the background region. A multi-scale convolutional network is used to process the labeled sequence of the background region. Specifically, this multi-scale convolutional network takes the labeled sequence of the background region as input, and gradually reduces the resolution of the feature map through layer-by-layer convolution and pooling operations, finally learning a two-dimensional vector with a sampling stride of 16. Then, this two-dimensional vector is flattened into a one-dimensional vector, which serves as the final background query vector. The reason for using a multi-scale convolutional network to downsample the labeled sequence of the background region is based on the following two considerations: First, the redundancy of background information, that is, the background region of an image usually contains a lot of redundant information, such as repeated textures, similar colors, etc. Therefore, fewer labels can be used to effectively represent the information of the background region without losing too much discriminative ability. Second, reducing computational complexity: since the computational complexity of attention in visual Transformers is quadratically proportional to the number of input labels, reducing the number of labels in the background region can significantly reduce the computational load of the model and improve tracking speed and efficiency.

[0050] By designing separate target query vectors and background query vectors and employing different modeling strategies, this invention effectively decouples the learning of temporal information about the target and the background. The target query vector focuses on aggregating the trajectory and appearance change information of the target, while the background query vector minimizes computation while ensuring the representational ability of the background information. This differentiated design is one of the key factors enabling this invention to effectively cope with complex background interference.

[0051] Introducing attention computation for time-series query updates

[0052] This invention uses the commonly used two-dimensional visual Transformer network as the basic architecture of the backbone network. However, in order to accommodate the introduction of temporal query vectors and realize the autoregressive update of query vectors between frames, the key innovation of this invention lies in extending the original spatial two-dimensional attention calculation method and designing a new attention calculation mechanism that introduces temporal query updates.

[0053] Traditional Transformer-based target tracking methods typically use single-frame image pairs as input, and the attention calculation process of their backbone network can be summarized as follows: , in, Image markers representing template frames, Image markers representing search frames, This represents a traditional two-dimensional attention computation operation. The visual features are labeled using attention computation. This attention computation method can only model visual similarity within a single frame and cannot capture the temporal correlation information between consecutive frames in a video sequence.

[0054] like Figure 2 As shown, to solve the above problems, this invention expands the input to a multi-frame video sequence and propagates the temporal query vector between consecutive frames in an autoregressive manner, thereby implicitly modeling temporal information. Specifically, for the The attention calculation method of this invention can be expressed as follows:

[0055] ,

[0056] ,

[0057] in, For the first Template frame marker for the frame, For the first Frame search frame markers, It is a trainable linear projection layer used to map image labels to a query, key, and value space. The first Visual queries, key and value tags for frames; The visual feature labels obtained by attention calculation, These are the target query vector and the background query vector, respectively.

[0058] The key to this invention lies in introducing a target query vector in each video frame. and a background query vector This is used to compress and store target and background information in temporally densely sampled video sequences. In the current frame, visual feature mapping is performed using an attention mechanism. and time-series query vector Perform association calculations. More importantly, update the target query vector in an autoregressive manner. and background query vector From the Frame propagation to the first Frame, and compare it with the first Visual queries, key and value tags for frames Together as the first The input for frame attention calculation. The specific process is as follows: Figure 2 As shown.

[0059] In this target tracking paradigm based on time-series query updates, the present invention can update the target query vector. and background query vector Serving as a cue for attention calculation in the next frame—that is, using past information to guide future reasoning—aligns with the intuition of target tracking in continuous video sequences. Furthermore, the target query vector... and background query vector The propagation is equivalent to implicitly propagating the location and appearance information of the target and the background, which can complement and enhance the image visual features provided by the template frame and the search frame, thereby effectively improving the performance of target tracking.

[0060] The long-term target tracking method based on time-series query propagation of this invention includes the following steps:

[0061] Step 1: Video sequence segmentation and patch embedding: Obtain a set of template frames including multiple frames. and multi-frame search frame set The video sequence is divided into frames, each frame is flattened into a series of image blocks. Then, each image block is flattened into a one-dimensional vector. This operation transforms image information into sequence data that is easier for models like the Transformer to process. A trainable linear projection network is used. The flattened template frame image block and the search frame image block are mapped to a high-dimensional latent space to obtain the board patch embedding. and search patch embedding The role of linear projection networks is to transform raw pixel information into more discriminative feature representations.

[0062] Trainable linear projection network It can be a linear function, and the dimension of the high-dimensional latent space can be 384.

[0063] Step 2: Adding Location Embedding: To enable the Transformer model to perceive the location information of the patch in the original image, the template patch embedding obtained in Step 1 is added. and search patch embedding Add learnable position embeddings respectively and Location embedding can encode the spatial location of a patch in an image, allowing the model to consider spatial relationships when performing attention calculations.

[0064] Adding positional embeddings to template patch embeddings and search patch embeddings involves element-wise addition of the template patch embedding and search patch embedding to the positional embedding. The positional embedding is a learnable vector, obtained by defining a tensor and then having a deep learning network autonomously learn the positional encoding information.

[0065] Step 3: Initial Query and Transformer Interaction: Design two learnable query vectors, namely the target query vector. Background query vector The target query vector aims to learn and represent the salient features of the target object, while the background query vector aims to learn and represent the features of the background region. The first frame template patch obtained in step two is then embedded... Search patch embedding With the target query vector of the design Background query vector Multiple visual Transformer layers are input together for attention computation. The attention computation is performed by considering the target query vector across multiple visual Transformer layers. and background query vector will be respectively with and Each patch embedding in the algorithm undergoes cross-attention computation to extract feature information relevant to the target and background. Simultaneously, Internal and The internal patch embeddings also perform self-attention computation to learn intra-frame contextual relationships.

[0066] The number of visual Transformer layers can be up to 24.

[0067] Step 4: Temporal Query Propagation and Transformer Iteration: The target query vector calculated by the visual Transformer layer in Step 3... Background query vector As the updated query vector. The updated target query vector. Background query vector Embedded template patch with subsequent frames and search patch embedding The inputs are fed into multiple visual Transformer layers for further attention calculation. The query is repeatedly updated and fed into the visual Transformer layers; that is, attention calculations are continuously performed using Transformer layers between consecutive video frames. This calculation process applies to template patch embedding and search patch embedding for frames other than the first and second frames. During this process, the target query vector... and background query vector It will continuously update and propagate during the interaction with image information in previous and subsequent frames, thereby implicitly learning and modeling the temporal dynamic changes of the target and background.

[0068] Step 5: Embed the search frame patch generated in Step 4 after multiple Transformer iterations. Input the prediction head to obtain the target's location coordinates. This prediction head consists of a fully connected layer used to embed the patch and map it to the target's location coordinate information, including the target's center point coordinates and width and height.

[0069] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0070] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A long-term target tracking method based on time-series query propagation, characterized in that, include: Step 1: Video sequence segmentation and patch embedding: Obtain a set of template frames including multiple frames. and multi-frame search frame set The video sequence frames are divided and flattened into a series of image patches. Then, each image patch is flattened into a one-dimensional vector, and a trainable linear projection network is used. The flattened template frame image patch and the search frame image patch are mapped to a high-dimensional latent space to obtain the template patch embedding. and search patch embedding ; Step 2: Add Position Embedding: Embed the template patch obtained in Step 1 and search patch embedding Add learnable position embeddings respectively and ; Step 3: Initial Query and Transformer Interaction: Designing the Target Query Vector Background query vector The target query vector aims to learn and represent the salient features of the target object, while the background query vector aims to learn and represent the features of the background region. The first frame template patch obtained in step two is then embedded. Search patch embedding With the target query vector of the design Background query vector Multiple visual Transformer layers are input together for attention calculation; among them, the target query vector... It is randomly initialized as a one-dimensional vector, the background query vector. The acquisition methods include: using a multi-scale convolutional network with the labeled sequence of the background region as input, gradually reducing the resolution of the feature map through layer-by-layer convolution and pooling operations, and finally learning a two-dimensional vector with a sampling stride of 16. Then, the two-dimensional vector is flattened into a one-dimensional vector as the final background query vector. Step 4: Time-series query propagation and Transformer iteration: Update the target query vector in Step 3. Background query vector Embedded template patch with subsequent frames and search patch embedding The input is fed into multiple visual Transformer layers, and attention is calculated again; the target query vector is repeated. Background query vector Update and input the visual Transformer layer, that is, continuously use the Transformer layer to perform attention calculations between consecutive video frames; Step 5: Embed the search frame patch generated in Step 4 after multiple Transformer iterations. Input the prediction header to obtain the target location coordinates.

2. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, Trainable linear projection network It is a linear function.

3. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, The dimension of the high-dimensional latent space is 384.

4. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, Adding positional embedding to template patch embedding and search patch embedding involves adding positional embedding to each template patch embedding and search patch embedding element by element.

5. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, Location embedding is a learnable location embedding vector. The way to obtain location embedding is to directly define a tensor and then have a deep learning network learn the location encoding information autonomously.

6. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, Target query vector in multiple visual Transformer layers and background query vector will be respectively with and Each patch embedding in the algorithm undergoes cross-attention computation to extract feature information relevant to the target and background; simultaneously... Internal and The internal patch embeddings also perform self-attention computation to learn intra-frame contextual relationships.

7. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, The number of multiple visual Transformer layers is 24.

8. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, The prediction head consists of a fully connected layer used to embed the patch into the target's location coordinate information.

9. The long-term target tracking method based on time-series query propagation according to claim 1, characterized in that, The target's location coordinates include the coordinates of its center point and its width and height.

Images