Single-target tracking method and device based on state space model and attention and medium

By adopting a hybrid architecture of state space model and Transformer in the field of single-target tracking, combining bidirectional state space model blocks and Transformer blocks, using timing tokens for cross-frame information propagation, the problem of computing burden based on Transformer tracker at high resolution is solved, and efficient and accurate target tracking is achieved.

CN120235912APending Publication Date: 2025-07-01NANJING UNIV

Patent Information

Application Number
CN202510314477.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-17
Publication Date
2025-07-01

AI Technical Summary

Technical Problem

Transformer-based trackers face the challenge of computing burden when expanding image resolution, while state space models are difficult to make significant progress in the single-target tracking field due to limited retrieval capabilities.

Method used

A tracker based on a hybrid architecture based on a state space model and Transformer is adopted to achieve efficient tracking of goals through hierarchical feature extraction, cross-frame information propagation, target enhancement and prediction head design. This architecture combines bidirectional state space model blocks and Transformer blocks to use timing tokens to propagate information across frames and enhances features through point multiplication.

Benefits of technology

Improves tracking accuracy and stability, reduces computing and memory consumption, and significantly improves the scalability of large-resolution video tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120235912A_ABST
    Figure CN120235912A_ABST
Patent Text Reader

Abstract

A single-target tracking method and device based on a state space model and attention and a medium are characterized in that firstly, hierarchical feature extraction is carried out, hierarchical modeling is carried out on a template and a search area through a mixed attention mechanism and the state space model, and a time sequence token is introduced into each frame at the final stage of hierarchical feature extraction to carry out target information aggregation; and aggregating the time sequence tokens by using a state space model to realize cross-frame information propagation, multiplying the obtained time sequence tokens by corresponding frame feature maps to serve as feature enhancement, and finally predicting a bounding box of a target object by a prediction head to realize target representation. According to the method, the state space model and the attention mechanism are combined for single-target tracking, especially target tracking under a large-resolution video, the state space model is introduced into the field of single-target tracking, the advantage of linear complexity of the state space model is brought into full play, meanwhile, the disadvantage of limited retrieval capacity of the state space model is made up, and high-resolution video tracking is achieved. And finally, the consumption of computing resources is reduced while accurate tracking is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, involving machine learning, computer vision, and visual object tracking technologies. A tracking system based on a hybrid architecture of a state space model and Transformer is proposed to real-time locate and track target objects in a video sequence, and it is a single-object tracking method, device, and medium based on a state space model and attention. Background Art

[0002] Visual object tracking is a core task in computer vision, whose purpose is to locate a target object in a video sequence based on the initial state of the target. Although traditional tracking methods have made some progress in addressing challenges such as target deformation, occlusion, and confusion with similar objects, they still face challenges in model efficiency when dealing with high-resolution data, including computational and memory burdens. Existing Transformer-based trackers have achieved remarkable success in global modeling capabilities, but due to the quadratic complexity of the attention mechanism, they face challenges in computational burden when expanding image resolution, which is crucial in spatial modeling and localization. Structured state space models exhibit the advantage of linear complexity in sequence modeling, but their retrieval and localization capabilities are limited by the finite number of hidden states. To overcome the limitations of these two types of models, some studies have proposed hybrid models that combine the attention mechanism with the state space model to achieve the optimal balance between effectiveness and efficiency.

[0003] Some existing technical solutions are as follows:

[0004] (1) Patent application CN119006529A, "A visible light-thermal infrared target tracking method based on robust spatio-temporal context modeling of a state space model". This method uses a state space model to perform multi-object tracking on multiple template frames and search frames of visible light-thermal infrared target pictures, and utilizes the linear long-sequence modeling ability of the visual state space model to embed the long-term cross-frame integration component and the short-term historical trajectory hint into the bidirectional visual state space model encoder (Mamba) to perform spatio-temporal context unified modeling of global appearance information and local motion information in a single-stream framework. However, this solution only conducts experiments on the light-thermal infrared image dataset, does not verify the target tracking effect on natural RGB images, and does not consider the problem of the limited retrieval ability of the pure state space model itself.

[0005] (2) Patent Application CN119089261A, "A Trajectory Prediction Method and System for State Space Aircraft Based on Wavelet Transform". Obtain the historical observation sequence of the original trajectory of the target aircraft; extract the frequency domain features of the historical observation sequence through discrete wavelet transform, and construct a trajectory prediction model that combines state space and Transformer encoder. Input the frequency domain features of the historical observation sequence into the trained trajectory prediction model to generate the frequency domain features of the future predicted trajectory, and restore the trajectory sequence of the aircraft through inverse wavelet transform of the frequency domain features of the future predicted trajectory to determine the prediction result. However, this solution only processes and predicts using the state space model and Transformer module on the trajectory frequency domain feature data, ignoring the appearance feature modeling of the objects in the image, and the application scenario is limited.

[0006] (3) Patent Application CN118537369A, "A Single Object Tracking Network Based on the Combination of Vmamba and Transformer", provides a single object tracking network based on the combination of Vmamba (Visual mamba, visual state space model) and Transformer. The single object tracking network includes a feature extraction network, an attention feature fusion network based on the AiA (Attention in Attention) module, a tracking head network, and an intermediate frame update network. The feature extraction network is used to extract image features from the initial frame, intermediate frame, and search frame; the attention feature fusion network based on the AiA module is used to perform self-attention and cross-attention calculations on the image features to generate an enhanced feature map; the tracking head network is used to predict the coordinate position of the target bounding box on the feature map and return the search frame; the intermediate frame update network is used to add the newly detected target to the intermediate frame to obtain the latest feature map of the target and improve the tracking performance of the algorithm. This invention introduces Vmamba into the field of visual tracking to solve the problem that the CNN as the backbone network lacks a global receptive field, but there are still the following deficiencies: 1. The state space inside the backbone network Vmamba needs to be scanned in four directions, which is relatively inefficient. 2. As a pure state space model, the backbone network Vmamba has a poor modeling effect and needs to rely on the subsequent AiA module to introduce an attention mechanism layer for compensation, resulting in the risk of insufficient modeling of the backbone network. 3. The way of its feature enhancement is AiA, which is essentially a multi-layer attention module, and this way is relatively bloated and not conducive to the lightweight of the model. Summary of the Invention

[0007] The technical problem to be solved by the present invention is as follows: As described in the background art, due to the quadratic complexity problem of the attention mechanism, the tracker structure based on Transformer faces challenges in computational burden when expanding the image resolution; the state space model theoretically has the advantage of linear complexity, but due to the limited number of states, its retrieval ability is poor. There is little research on the state space model in the field of single-object tracking. How to introduce the state space model into the field of single-object tracking, give full play to its advantage of linear complexity while making up for the disadvantage of limited retrieval ability, and reduce the consumption of computing resources while achieving accurate tracking still needs to be explored.

[0008] The technical solution of the present invention is: A single-object tracking method based on the state space model and attention, constructing a hybrid architecture tracker based on the state space model and Transformer, using the first-frame annotation as the input template and the subsequent video frames as the search area to perform single-object tracking in the video. The tracked target in the subsequent video frames is used as the online template. The hybrid architecture tracker includes hierarchical feature extraction, cross-frame information propagation, target enhancement, and a prediction head. Cross-frame information propagation is introduced in the last stage of hierarchical feature extraction. Before entering the prediction head for target prediction, the temporal tokens obtained by cross-frame information propagation are used for target enhancement:

[0009] Hierarchical feature extraction: Through the combination of a bidirectional state space model block and a Transformer block, multi-level feature extraction is performed on the input picture sequence, gradually extracting the visual features of the target from low resolution to high resolution in stages. The bidirectional state space model block serializes the template and the search area using a variety of sequence rearrangement strategies, and performs reverse serialization on the image based on the local flipping mechanism. Bidirectional modeling is performed according to the results of serialization and reverse serialization; the Transformer block uses an asymmetric attention mechanism during the bidirectional modeling process, allowing information to flow from the template area to the search area, but blocking the information flow from the template area to the search area.

[0010] Cross-frame information propagation: For each frame, the input template and search area sequences are converted into token sequences through downsampling and block processing. In the last stage of hierarchical feature extraction, a temporal token is assigned to each frame and added to the token sequence. The temporal token is used to store the feature information and temporal information of the target in that frame. The token sequence with the added temporal token is modeled by the combination of the bidirectional state space model block and the Transformer block. The evolution of the target state over time is reflected by the update of the temporal token for each frame. Finally, the temporal tokens of each frame are extracted and modeled using the temporal state space model to achieve cross-frame propagation of target information;

[0011] Target Enhancement: Temporal tokens are modeled using a temporal state space model and fused with the feature map of the current frame, that is, the temporal tokens of this frame are multiplied by the feature map of this frame to enhance the target features of this frame of image;

[0012] Prediction Head: The prediction head predicts the bounding box of the target object from multi-level features. The prediction head includes multiple convolutional layers, which process and fuse the multi-scale feature maps enhanced by the target to achieve target representation for outputting the tracking results.

[0013] Furthermore, in the inference stage, an online template update mechanism and a score prediction module are adopted to dynamically update the template according to the appearance change of the target in the video. Among them, the target prediction of each frame outputs a confidence score through the score prediction module. The target region corresponding to the frame with the highest confidence score is used every fixed number of frames, and the online template is updated in a first-in-first-out (FIFO) manner.

[0014] Furthermore, based on the property of unidirectional modeling of the state space model, the bidirectional state space model block uses multiple sequence rearrangement strategies, including sequential rearrangement - overall flip, sequential rearrangement - separate flip, and interleaved rearrangement - overall flip:

[0015] Sequential rearrangement - overall flip: The input sequence is first rearranged in order, and then the whole sequence is flipped as a whole;

[0016] Sequential rearrangement - separate flip: While keeping the sequence order unchanged, each part is flipped frame by frame separately;

[0017] Interleaved rearrangement - overall flip: The tokens of the input sequence are interleaved and rearranged according to the same central position in different frames, while keeping the positions of the background tokens around the search area unchanged.

[0018] Furthermore, the asymmetric attention mechanism includes: for the features of the template region, self-attention calculation is performed only using the features of the template region itself; for the features of the search region, cross-attention calculation is performed using the combination of the features of the template region and the search region as the key values.

[0019] Furthermore, the hybrid architecture tracker is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through remote call or local call to achieve single-object tracking of a video.

[0020] The present invention also provides an electronic device, which includes a processor and a memory. At least one instruction or at least one program segment is stored in the memory, and the at least one instruction or the at least one program segment is loaded and executed by the processor to implement the above-mentioned single-object tracking method based on the state space model and attention, and realize single-object tracking based on the input template and video.

[0021] The present invention also provides a computer-readable storage medium storing a computer-executable program, which, when executed by a processor, implements the above-mentioned single-object tracking method based on the state space model and attention.

[0022] The present invention proposes a hybrid architecture tracker based on the state space model and Transformer, aiming to transmit target features through the scanning mechanism of the state space model. By using the state space model to model temporal tokens and using temporal information to enhance target features, this framework can directly achieve target aggregation and efficient feature transmission across frames.

[0023] MambaVision is a picture pre-training representation model that can only handle single-image classification problems. Based on MambaVision, the present invention adapts and improves it in the field of single-object tracking, adapts various serialization methods to the field of single-object tracking, and proposes an improved method for cross-frame information propagation and target enhancement. Compared with the solution of CN118537369A, the present invention only uses scanning modeling in 2 directions, with higher efficiency. The feature extraction backbone network used in the present invention is a hybrid structure of the state space model and the Transformer model, with sufficient backbone network modeling and better effects. In addition, the present invention uses temporal tokens and the temporal state space model to accelerate the flow of target information between frames, improving the stability of tracking. Finally, the method of dot multiplication is used for feature enhancement, which is more lightweight.

[0024] A large number of experiments show that the tracker proposed by the present invention is superior to the first-generation single-stream Transformer-based tracker in terms of performance and memory consumption at the same resolution, especially in terms of scalability at large resolutions. Compared with the prior art, the advantages of the present invention are as follows:

[0025] 1) Improve tracking accuracy: Through the design of temporal tokens and the temporal state space model, the model can more effectively transmit target information between frames, improving the continuity and accuracy of tracking. Through the state space model scanning mechanism and target enhancement technology, the tracking accuracy of the target is improved, especially in high-resolution videos.

[0026] 2) Reduce computational and memory consumption: Utilize the linear complexity characteristics of the state space model to alleviate the computational and memory consumption during the processing of large-resolution videos.

[0027] 3.) Scalability in improving resolution: By combining multi-scale feature fusion technology, the model of the present invention can capture the features of the target more comprehensively, improving the accuracy of tracking and the adaptability to different scale changes. When expanding the input resolution, the present invention can significantly improve the tracking performance while maintaining low computational consumption, showing good scalability. Description of the Drawings

[0028] Figure 1 It is a modeling method of various bidirectional serialization methods based on a bidirectional state space model implemented by the present invention.

[0029] Figure 2 It is a network schematic diagram of a hybrid architecture tracker of the present invention based on a state space model and Transformer.

[0030] Figure 3 It is the difference in training video memory overhead of the model in the method of the present invention compared with MixCvT and MixViT at different resolutions. Detailed Embodiment

[0031] The present invention provides an efficient visual object tracking framework TrackMamba that combines a state space model and Transformer. This hybrid architecture tracker realizes the efficient transmission and aggregation of target features through the combination of a state space model block and an attention block. The present invention simultaneously uses a bidirectional state space model and an asymmetric attention mechanism to model the object features in the video frame sequence to reduce the overhead on high-resolution videos. And temporal tokens are introduced for cross-frame information propagation, and the feature maps are enhanced using temporal tokens. The present invention is efficiently modeled based on a bidirectional state space model and an asymmetric attention mechanism. Specifically, the object features in the video frame sequence are modeled using a state space model, where the state space model processes the input sequence using a bidirectional serialization modeling method, and the features of the template region and the search region are processed using an asymmetric attention mechanism, allowing the features of the template region to flow to the search region while preventing the features of the search region from flowing to the template region.

[0032] The present invention aims to solve the computational and memory consumption problems faced by existing single-stream trackers based on Transformer in large-resolution video tracking tasks while maintaining tracking accuracy. First, the present invention enhances the model's context awareness ability for the input sequence and maintains linear complexity by introducing the core operation of the structured state space model: the selective scanning mechanism, which is crucial for processing high-resolution videos. The present invention proposes a tracking framework of a hybrid state space model-Transformer that utilizes additional target enhancement techniques to achieve the aggregation and cross-frame propagation of target features through temporal tokens and a temporal state space model, improving the final tracking effect. AsFigure 2 As shown, at the end of hierarchical modeling, the present invention adopts cross-frame information propagation. Before entering the prediction head, the temporal tokens obtained by cross-frame information propagation are used for target enhancement. The implementation of the present invention will be specifically described below.

[0033] Hierarchical feature extraction: Corresponding to Figure 2 Stages 1, 2, 3, and 4 shown in the figure, the size of the feature map is gradually reduced layer by layer, while the number of channels increases, and the visual features of the target are gradually extracted in stages from low resolution to high resolution.

[0034] The present invention realizes hierarchical feature extraction based on a hybrid attention mechanism and a state space model. The TrackMamba framework of the present invention realizes feature extraction based on a hybrid attention mechanism and a state space model. Its backbone network includes interleaved bidirectional state space layers and asymmetric attention layers, combining the linear complexity advantage of the structured state space model and the global modeling ability of the attention mechanism to achieve efficient feature extraction and target search integration. Among them, the bidirectional state space layer includes state space models in the forward and backward directions. By using the bidirectional information flow method, the state space can more comprehensively model the features of the target and the image; the asymmetric attention layer means that for the template region, the key values corresponding to its attention query only include the template region itself, while for the search region, the key values corresponding to its attention query include both the template region and the search region. This asymmetric attention method can block the information flow from the search region to the template region to keep the template pure.

[0035] Perform multi-level feature extraction on the input picture sequence. Through the combination of a bidirectional state space model block and a Transformer block, the visual features of the target are gradually extracted from low resolution to high resolution. Among them, based on the property of unidirectional modeling of the state space model, the bidirectional state space model block uses a variety of sequence rearrangement strategies to serialize the template and search regions, and performs reverse serialization on the image based on the local flipping mechanism. Finally, bidirectional modeling is performed according to the results of serialization and reverse serialization to improve the efficiency and accuracy of information flow. The Transformer block uses an asymmetric attention mechanism, allowing information flow from the template region to the search region during the bidirectional modeling process, but blocking the information flow from the template region to the search region; the above bidirectional state space model block and attention block can gradually extract the visual features of the target from low resolution to high resolution, forming a hierarchical feature representation. Due to the introduction of the state space model, the present invention can model more efficiently under high-resolution picture sequences.

[0036] The above-mentioned multiple bidirectional serialization methods based on the state space model for modeling are as Figure 2 The serialization and splicing shown in Stage 3 specifically include the following steps:

[0037] The template and search area sequences input by S1 are first downsampled and segmented, and then converted into token sequences to prepare for subsequent scanning and processing. The token sequences serve as the input to the state space model, which gradually captures the spatio-temporal patterns in the video by processing these token sequences.

[0038] S2. As Figure 1 shown, based on the property of unidirectional modeling of the state space model, the bidirectional state space model block uses multiple sequence rearrangement strategies, including (A) sequential rearrangement - overall flipping, (B) sequential rearrangement - separate flipping, and (C) interleaved rearrangement - overall flipping. The input sequence is rearranged and flipped accordingly according to the selected scanning mode. Among them, (A) sequential rearrangement - overall flipping means that the input sequence is first rearranged in order and then the whole sequence is flipped as a whole. This mode is simple and direct, but the flow of target information may be disturbed due to the overall flipping during processing, especially when the search area contains a large amount of background information; (B) sequential rearrangement - separate flipping means that while keeping the sequence order unchanged, each part is flipped separately. This flipping method avoids the problem of target information flow caused by overall flipping, ensures the correct transmission of target information during scanning, and reduces the interference of background information. (C) interleaved rearrangement - overall flipping means that the tokens of the input sequence are interleaved and rearranged according to the same central position in different frames, while keeping the positions of the background tokens around the search area unchanged. The spliced sequence after this flipping method is directly used for inter-frame modeling, which is beneficial for direct inter-frame target information transmission, but will damage the intra-frame spatial information modeling. After experimental comparison, the embodiment of the present invention selects (B) sequential rearrangement - separate flipping as the final rearrangement scheme.

[0039] S3. The sequences after rearrangement and flipping are respectively input into the forward state space model and the backward state space model for processing. The forward state space model is responsible for the information flow from the template to the search area, while the backward state space model is responsible for the reverse information flow. The design of the bidirectional state space model enables the TrackMamba block to effectively process the input sequence, improves the accuracy of target tracking, and makes it have better adaptability and robustness in different tracking scenarios.

[0040] S4. The results processed by the forward and backward state space models are flipped back to the original order, added and rearranged to obtain the final target-related information for subsequent target localization and tracking.

[0041] Cross-frame information propagation: In the last stage of hierarchical feature extraction, corresponding Figure 2In stage 4, a temporal token is inserted into the token sequences of the template and the search region for each frame. Then, as in stage 3, it is modeled by the combination of the bidirectional state space model block and the Transformer block. Due to the global modeling nature of the bidirectional state space model and the Transformer model, the temporal tokens will obtain the overall temporal information. These temporal tokens are separately extracted and then further interacted through a temporal state space model, enabling the information to be propagated across frames.

[0042] The cross-frame information propagation technology aims to assign a temporal token to each frame to represent the target information in a single frame, and uses the state evolution mechanism of the state space model itself for modeling to achieve the cross-frame propagation of the target information. The cross-frame propagated target information enables the model to accumulate and utilize the target information in historical frames, thereby enhancing the model's long-term memory and recognition ability of the target, especially in the case where the appearance of the target changes or is occluded. The specific implementation is as follows:

[0043] 1. Generate temporal tokens for each frame: When processing a video sequence, the present invention generates a temporal token for each frame and inserts it at the head of the token sequence of each frame. This token, as a vector, is used to aggregate and store the feature information of the target in this frame.

[0044] 2. Model the target information of the temporal tokens using context: The temporal tokens and the tokens corresponding to the original frames are sent together to the bidirectional state space model layer and the attention layer for unified modeling. The temporal tokens are updated through the global nature of the bidirectional state space and the attention mechanism to simulate the evolution of the target information over time in the video sequence.

[0045] 3. Achieve cross-frame propagation of the target information: The temporal tokens at the head of each frame are extracted and further modeled using the temporal state space model. In this embodiment, the temporal state space model preferably adopts a lightweight bidirectional temporal state space model, which includes 3 layers of bidirectional temporal state space layers. Through the temporal state space model, the temporal tokens storing the target information can directly interact using the state evolution mechanism of the state space model, thereby achieving the direct cross-frame propagation of the target information. This mechanism improves the stability of the model in modeling the target in high-resolution video scenarios.

[0046] Target enhancement: Before entering the prediction head, the temporal tokens obtained by cross-frame information propagation are used for target feature enhancement.

[0047] Target feature enhancement multiplies the timing token that carries the target timing information modeled by the temporal state space model with the feature map of the current target to enhance the feature. The timing token proposed in the cross-frame information propagation carries the timing information of the target and is used to fuse with the feature map of the current frame to enhance the target features of the current frame. Specifically, the fusion of the timing token and the feature map is achieved through a simple multiplication operation, that is, the timing token of the current frame is multiplied by the feature map of the current frame, thereby integrating the temporal information into the spatial features of the current frame. The feature map after the fusion operation not only contains the spatial information of the current frame, but also integrates the temporal information across frames, which makes the feature map more robust and accurate in target recognition and tracking.

[0048] Prediction head: Use the bounding box prediction head to predict the bounding box of the target object from multi-level features. The multi-level features include Figure 2 The features of different scales obtained in the four stages shown include feature maps downsampled to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original resolution. The multi-scale features are gradually upsampled and fused using convolution, and finally a feature map with a resolution of 1 / 4 of the original resolution is obtained, and the position of the target box is predicted based on this feature map.

[0049] Furthermore, the present invention also includes online template updating and score prediction: using an online template updating mechanism and a score prediction module, the online template is dynamically updated according to the appearance changes of the target in the video, wherein the target prediction of each frame outputs a confidence score through the score prediction module, and the frame with the highest confidence score is used to correspond to the target area every fixed number of frames, and the online template is updated using a first-in-first-out FIFO method. Figure 2 As shown in the figure, the template is the real target corresponding area map in the first frame; the online template is the target corresponding area map dynamically updated in the subsequent frames, and they belong to the same tracking target. The tracking performance of target tracking is further improved by dynamically updating the online template.

[0050] As shown in Table 1, the three rearrangement flipping strategies proposed in this invention are compared on the LaSOT dataset. The evaluation criteria include AUC (Area Under Curve), normalized precision P NORM and precision P. We found that (C) interlaced rearrangement-overall flipping strategy has the worst effect because it destroys the continuity of the sequence within the frame; (A) sequential rearrangement-overall flipping strategy has a decent effect, but the overall flipping operation introduces unnecessary information flow from the search area to the template area, while (B) sequential rearrangement-separate flipping strategy has the best effect because it cuts off unnecessary information flow from the search area to the template area. The present invention selects (B) sequential rearrangement-separate flipping strategy as the final solution and performs high-resolution expansion based on it.

[0051] Table 1: Comparison of the effects of different scanning methods at 256 resolution on LaSOT

[0052]

[0053] As shown in Table 2, the model of the present invention was evaluated on three mainstream single-object tracking datasets, namely LaSOT, TrackingNet, and GOT-10k. The evaluation criteria include AUC (Area Under Curve), normalized precision P NORM and precision P, average overlapping rate AO (Average Overlapping), and success rate SR Threshold (Success Rate).

[0054] Compared with MixFormer, the representative of the first-generation single-stream Transformer tracker, whose backbone network has two versions, CvT and ViT, and the pre-training effects are both better than MambaVision. The embodiments of the present invention mainly compare the ViT version. As can be seen from Table 2, the model TrackMamba of the present invention is better than MixCvT and MixViT at the same resolution (256); comparing the resolution expansion effects of MixViT and TrackMamba, the experiments show that the model of the present invention is better than MixViT when the resolution is expanded to 384. When the resolution is expanded to 512, MixViT cannot continue to expand due to limited video memory, while TrackMamba successfully expands and achieves an improvement in effects.

[0055] Table 2: Comparison of the effects of different models on LaSOT, TrackingNet, and GOT-10K

[0056]

[0057]

[0058] Figure 3 Shows the difference in the training video memory overhead of the present invention compared with MixCvT and MixViT at different resolutions. When the resolution is expanded from 256 to 512, the video memory growth rates of MixCvT and MixViT are relatively fast, and memory overflow (<40G) will occur when processing 512 resolution. However, due to the introduction of the state space model, the model of the present invention significantly alleviates the video memory burden caused by the increase in resolution.

[0059] The present invention is implemented based on a computer program. The video object tracking model is implemented through at least one instruction or at least one program segment, and the at least one instruction or at least one program segment is used to be executed by a data processor through remote call or local call to implement the video object tracking task. The instructions or program codes for implementing the method of the present invention can be written in any combination of one or more programming languages. The instructions or program codes can be executed entirely on the data processor, partially on the processor, partially on the processor as an independent software package and partially on a remote device, or entirely on a remote device or server.

[0060] The efficient visual object tracking framework based on the state space model-Transformer proposed by the present invention can improve the efficiency and accuracy of large-resolution video object tracking tasks, and enhance the computing performance and memory utilization rate of the model when processing high-resolution video data. By combining the advantages of the state space model and Transformer, the present invention designs a novel hybrid architecture, enabling the object tracker to accurately track the object while maintaining a low computational complexity in high-resolution situations, which is particularly important for video analysis and monitoring systems. The present invention has a wide range of application scenarios in the field of computer vision, including but not limited to security monitoring, autonomous driving, sports event analysis, robot navigation, etc.

Claims

1. Single target tracking method based on state space model and attention, characterized by A hybrid architecture tracker based on the state space model and Transformer is constructed. The first frame annotation is used as the input template and the subsequent video frames are used as the search area to track a single target in the video. The target tracked in the subsequent video frames is used as the online template. The hybrid architecture tracker includes hierarchical feature extraction, cross-frame information propagation, target enhancement and prediction head. Cross-frame information propagation is introduced in the last stage of hierarchical feature extraction. Before entering the prediction head for target prediction, the temporal tokens obtained by cross-frame information propagation are used to enhance the target: Hierarchical feature extraction: Through the combination of the bidirectional state space model block and the Transformer block, multi-level feature extraction is performed on the input image sequence, and the visual features of the target are extracted in stages from low resolution to high resolution. The bidirectional state space model block uses a variety of sequence rearrangement strategies to serialize the template and search area, and reverse serializes the image based on the local flipping mechanism. Bidirectional modeling is performed based on the results of serialization and reverse serialization. The Transformer block uses an asymmetric attention mechanism in the bidirectional modeling process, allowing information from the template area to flow to the search area, but blocking the flow of information from the template area to the search area; Cross-frame information propagation: For each frame, the input template and search area sequence are converted into a token sequence through downsampling and block processing. In the last stage of hierarchical feature extraction, a timing token is assigned to each frame and added to the token sequence. The timing token is used to store the feature information and timing information of the target in the frame. The token sequence with the timing token is modeled by the combination of the bidirectional state space model block and the Transformer block. The evolution of the target state over time is reflected by updating the timing token of each frame. Finally, the timing token of each frame is extracted and modeled using the timing state space model to achieve cross-frame propagation of target information. Target enhancement: Use the time-series state space model to model the time-series token and merge it with the feature map of the current frame, that is, multiply the time-series token of this frame with the feature map of this frame to enhance the target features of the current frame; Prediction head: The prediction head predicts the bounding box of the target object from multi-level features. The prediction head includes multiple convolutional layers, which processes and fuses multi-scale feature maps enhanced by the target to achieve target representation for outputting tracking results.

2. The single target tracking method based on state space model and attention according to claim 1, characterized in that In the inference stage, an online template update mechanism and a score prediction module are used to dynamically update the template according to the appearance changes of the target in the video. The target prediction of each frame is outputted with a confidence score through the score prediction module. The frame with the highest confidence score is used to correspond to the target area every fixed number of frames, and the online template is updated using the first-in-first-out (FIFO) method.

3. The single target tracking method based on state space model and attention according to claim 1, Its characteristic is that the bidirectional state-space model block uses a variety of sequence rearrangement strategies based on the properties of unidirectional modeling of the state-space model, including sequential rearrangement-overall flip, sequential rearrangement-separate flip and staggered rearrangement-overall flip: Sequential rearrangement-overall flip: The input sequence is first rearranged in order, and then the entire sequence is flipped as a whole; Sequential rearrangement - flipping separately: while keeping the sequence order unchanged, each part is flipped frame by frame; Interleaved rearrangement-overall flip: The tokens of the input sequence are interleaved and rearranged according to the same center position in different frames, while the positions of the background tokens around the search area remain unchanged.

4. The single target tracking method based on state space model and attention according to claim 1, characterized in that The asymmetric attention mechanism includes: for the features of the template area, only the features of the template area itself are used for self-attention calculation; for the features of the search area, a combination of the features of the template area and the search area is used as a key value for cross-attention calculation.

5. The single target tracking method based on state space model and attention according to claim 1, characterized in that The temporal state space model is a lightweight bidirectional state space model.

6. The single target tracking method based on state space model and attention according to claim 1, characterized in that The hybrid architecture tracker is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through a remote call or a local call to achieve single target tracking for the video.

7. An electronic device, characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the single target tracking method based on state space model and attention as described in any one of claims 1 to 6, and to implement single target tracking based on input templates and videos.

8. A computer-readable storage medium, characterized in that A computer executable program is stored, and when the program is executed by a processor, the single target tracking method based on a state space model and attention as described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Single-target tracking network based on combination of Vmamba and Transform

    CN118537369A

  • Visible light-thermal infrared target tracking method based on robust spatio-temporal context modeling of state space model

    CN119006529A

  • Wavelet transform-based state spacecraft trajectory prediction method and system

    CN119089261A

Cited By

  • Robot hand-eye calibration method

    CN120620225A

  • Tracking result generation method and device based on YOLO and MixFormer models, equipment and medium

    CN121190524A