Space-time single-target tracking method based on Mama double prompts

Abstract

A space-time single-target tracking method based on Mama double prompts belongs to the field of visual target tracking, solves the problems of limited target space-time information, single-stage representation and the like in an existing tracking method, and comprises the following steps: firstly, introducing explicit and implicit prompts to enhance the dynamic representation capability of a target in the time dimension; then, the optimized explicit prompt and the updated search features are transmitted stage by stage, and multi-stage progressive spatial-temporal feature enhancement is achieved; and finally, network parameters are constrained by adopting a joint optimization strategy of cross entropy loss, generalized intersection-to-parallel ratio loss and L1 loss, so that optimal model parameters suitable for a high-precision single-target tracking task can be obtained. The visual target tracking method has good generalization ability and robustness, can adapt to various visual target tracking tasks, shows high adaptability to various challenges in different scenes, and provides an efficient and reliable solution for complex visual target tracking requirements in the fields of automatic driving, video security and protection and the like.

Description

Technical Field

[0001] This invention belongs to the field of visual target tracking, specifically relating to a spatiotemporal single target tracking method based on Mamba dual cues. Background Technology

[0002] In the fields of autonomous driving, human-computer interaction, and video surveillance, visual target tracking faces many challenges, such as interference from similar objects, occlusion, and motion blur. These place higher demands on the robustness and accuracy of tracking algorithms. Traditional trackers typically rely on the initial appearance of the target to model the template and search area, overemphasizing static appearance features while ignoring the inherent temporal information in the video sequence; for example... Figure 1 As shown in (a), the update-based tracker enhances the network's adaptability by updating the target's appearance features, improving the understanding of changes in the target's appearance to some extent. However, it is prone to error accumulation, thus reducing overall tracking performance. Figure 1 As shown in (b), cue-based trackers capture temporal features and transmit them through video sequences using a small number of tags, representing a significant improvement over previous work. However, they typically use spatiotemporal features in the final stage of the network, ignoring the dynamic evolution of spatiotemporal features at different stages and limiting the ability to model temporal context. Furthermore, cue-based trackers often focus context modeling on a limited number of tags, resulting in insufficient spatiotemporal content information transmitted. Mamba-based trackers merely use Mamba as a replacement for attention mechanisms, remaining confined to traditional spatiotemporal modeling paradigms and failing to fully utilize its autoregressive modeling capabilities. Meanwhile, attention mechanisms still possess unique advantages in feature modeling. Utilizing the complementary strengths of Mamba and attention mechanisms, while fully leveraging spatiotemporal information from multiple stages of the network, provides new insights for tracker design. Summary of the Invention

[0003] The main objective of this invention is to address the limitation of spatiotemporal information flow of targets within a fixed length or a single stage in visual target tracking. A spatiotemporal single-target tracking method based on Mamba dual cues is proposed. Explicit and implicit cues are jointly updated via Mamba with the features of the current search region. This process includes a context encoder and cue Mamba modules. By leveraging Mamba's sequence modeling capabilities and appearance modeling capabilities through its attention mechanism, the tracker can effectively capture and utilize multi-level spatiotemporal dependencies, thereby achieving target tracking that is both robust and real-time.

[0004] The design concept of this invention addresses the limitations of existing visual target tracking methods, such as limited spatiotemporal information and single-stage representation, by proposing a dual-cue tracking method based on Mamba. The core concept lies in effectively leveraging the complementary advantages of Mamba and attention mechanisms to construct a target tracking framework that balances performance and accuracy. The Transformer visual backbone network serves as a unified method for target appearance feature extraction and interaction. By dividing the backbone network into multiple stages and introducing a Mamba module to each stage, it further supplements the missing temporal information in each stage, solving the problem of limited temporal information in long sequences in existing methods. In each stage, explicit and implicit cues are jointly updated via Mamba along with the current search features, enabling the model to effectively capture dynamic changes in the target. The updated key information is stored in the hidden states of Mamba to construct a temporally coherent contextual representation. This provides an innovative solution for scenarios with high real-time and accuracy requirements, such as human-computer interaction, autonomous driving, and video security. It exhibits stronger robustness, especially in complex scenarios, bringing significant performance improvements to visual target tracking tasks and demonstrating broad application prospects.

[0005] This invention is achieved through the following technical solution: a spatiotemporal single-target tracking method based on Mamba dual cues. First, this invention introduces explicit and implicit cues to enhance the cross-temporal modeling of target dynamics, addressing existing problems through a context-aware strategy of global and local spatiotemporal cues. The original Transformer network is decomposed into multiple stages, each equipped with a cue Mamba module to handle different levels of spatiotemporal cue information. In each stage, explicit cues are jointly modeled with image features through an attention mechanism to capture appearance changes in the current frame. These are then fused with implicit cues from previous time steps and search region features from the cue Mamba module. Utilizing Mamba's unique hidden state update strategy, the model stores and updates key spatiotemporal historical representations, enabling implicit cues to encode long-term dynamics. The optimized explicit cues and updated search features are then passed to the next stage, achieving progressive spatiotemporal enhancement through multi-stage fusion. Compared to previous methods relying on a single time cue or historical feature, this method provides more comprehensive spatiotemporal contextual cues, improving the tracker's understanding of target changes. During training, the network parameters are optimized by combining FocalLoss, GIoU Loss, and L1 Loss to obtain the optimal network parameters for target tracking. Specifically, this includes the following steps:

[0006] S1. First, collect several template images and a single search region image respectively. Use an image serialization processing strategy to perform progressive downsampling to obtain the feature sequences of the template images and the search region images respectively, and add corresponding display prompts for each search region image. Then, concatenate all image feature sequences and display prompts according to the merged width and height dimensions to form joint features.

[0007] S2. Input the joint features obtained in step S1 into the first prompt encoder. Use the first prompt encoder to perform spatiotemporal modeling of the joint features in several Transformer blocks. The spatiotemporal modeling includes appearance feature modeling and time feature modeling. Separate the obtained spatiotemporal modeling results into features of the search area and display time features, and use them as input to the prompt Mamba module.

[0008] S3. First, the features and temporal features separated in step S2 are combined with the staged implicit cues, and then the long-term spatiotemporal context information is aggregated in Mamba, and the features, temporal features and implicit cues are updated. Then, the updated features and temporal features are residually connected with the input of Mamba to obtain the secondary spatiotemporal modeling result. Finally, the secondary spatiotemporal modeling result is used as the input of the second cue encoder to perform secondary spatiotemporal modeling.

[0009] S4. Repeat steps S2~S3, use the display prompts output by the last prompt encoder to weight the search area features, input the weighted search area features into the tracking head, and predict the target position.

[0010] S5. The model parameters are optimized by using a joint loss function of Focal Loss (cross-entropy loss), GIoU Loss and L1 Loss to obtain the optimal model parameters for target tracking, thus completing spatiotemporal single target tracking based on Mamba dual cueing.

[0011] Further, step S1 includes the following steps:

[0012] S1-1. Acquire several template images Z and a single search region image X, respectively. , Processed according to the image serialization method Image patches of varying sizes are embedded, and then the original image is transformed through a two-stage linear layer merging process. Image patch embedding of size;

[0013] S1-2, will Image patch embedding of different sizes is reshaped along the spatial dimension to form a serialized template marker. and search area markers , , ;in, Indicates the number of input templates; Template tag The length of the marker, , The height of the template image. The width of the template image; Indicates the length of the marker in the search area. , The height of the search region image, The width of the search area image; D is the preset number of channels;

[0014] S1-3. First, mark each search area. Assign an additional learnable tag As a display prompt, Display prompt Used to capture target changes over time at low cost, where t represents the time step of the current frame; then, in the template tag Search area markers With display prompts Learnable location embeddings are added to each element to enhance the corresponding spatial information; finally, display prompts with location embeddings are added. Template tags With search area markers Cascade along spatial dimensions to form initial joint features for:

[0015] (1)

[0016] In equation (1), L represents the length of the joint feature. D represents the preset number of channels; This indicates a cascading operation.

[0017] Further, step S2 includes the following steps:

[0018] S2-1. First, the Transformer-based backbone network is divided into N stages, and M Transformer blocks are assigned to each stage. The joint feature of each Transformer block is: Then, an independent prompting Mamba module is assigned after the Transformer block of each stage, while no prompting Mamba module is assigned after the Transformer block of the final stage, forming N prompting encoders.

[0019] S2-2, In the N cue encoders formed in step S2-1, the joint features in each Transformer block are... Perform linear projections separately to obtain the following results: Matrix, computed using multi-head self-attention The matrix is:

[0020] (2)

[0021] (3)

[0022] In equations (2) and (3), The matrices represent the query matrix, the queried matrix, and the weight matrix, respectively. It is a linear projection function; For multi-head self-attention functions; The result of attention calculation;

[0023] S2-3, Apply the attention calculation results obtained in step S2-2 With joint features Perform residual joins to obtain residual results. for:

[0024] (4)

[0025] Then, the residual results The input is fed into a feedforward neural network (FFN) for further processing, yielding the output of a single Transformer block. for:

[0026] (5)

[0027] In equations (4) and (5), For layer normalization processing; This is the residual result; The result of attention calculation; It is a feedforward neural network;

[0028] Finally, the Transformer blocks in the first prompt encoder are overlaid to form the Transformer block output. for:

[0029] (6)

[0030] In equation (6), For the Transformer block stacking operation in the first prompt encoder, M is the number of Transformer blocks allocated in each stage, and i takes the values ​​1, 2, 3, ..., M;

[0031] S2-4. First, the output of the Transformer block stacking operation. Decoupling was performed to obtain the time characteristics of the first stage. Template features and search area features for:

[0032] ; (7)

[0033] In equation (7), () indicates a decoupling operation;

[0034] Then, the time characteristics of the first stage and search area features By cascading, the spatiotemporal features related to the target can be obtained. Spatiotemporal characteristics As a prompt for the Mamba module input , The spatiotemporal features for:

[0035] ; (8)

[0036] In equation (8), Cascade Operations

[0037] Furthermore, the prompting Mamba module includes a left branch and a right branch. In the prompting Mamba module, the spatiotemporal characteristics of the input... Following the standard Mamba architecture, step S3 is processed through two parallel branches, a left branch and a right branch, to simulate long-term target dynamics, including the following steps:

[0038] S3-1, In the right-hand branch:

[0039] First, linear projection is used to extend the spatiotemporal features. Channel dimension :

[0040] (9)

[0041] In equation (9), It is a linear projection function;

[0042] Then, the channel dimensions are enhanced through convolution operations. The local spatial dependence is then activated using the SiLU activation function to generate locally enhanced spatiotemporal features. for:

[0043] (10)

[0044] In equation (10), Causal convolution function;

[0045] Finally, the spatiotemporal features of local enhancement will be used. Combined with from As input to the State-Space Model (SSM), the long-term context information is captured and updated through the State-Space Model to generate the right-hand branch result. And updated implicit hints for:

[0046] (11)

[0047] In equation (11), Calculations are performed for the state-space model;

[0048] S3-2. A similar processing flow is also applied in the left-hand branch. In the left-hand branch:

[0049] First, a linear layer is used to expand the spatiotemporal features. Channel dimension :

[0050] (12)

[0051] In equation (12), It is a linear projection function;

[0052] Then, the SiLU activation function is used to activate the channel dimension. Generate left branch results for:

[0053] (13)

[0054] In equation (13),

[0055] S3-3, Results of the right-side branch and the results of the left branch The weighted sum is obtained by performing Hadamard multiplication operations together, and then the weighted result is reduced in dimensionality through linear projection to obtain the dimensionality-reduced output. for:

[0056] (14)

[0057] In equation (14), This is the result of the right-hand branch; This is the result of the left branch; Linear projection function; This represents the Hadamard multiplication operation;

[0058] S3-4. Dimensional Reduction Output Results Spatiotemporal characteristics of input to the Mamba module. The result of the quadratic spatiotemporal modeling (indicated by the output of the Mamba module) is as follows:

[0059] (15)

[0060] S3-5, by prompting the Mamba module's operation, effectively integrates current observations and historical dynamic signals, thereby converging the complete spatiotemporal content of the target. Therefore, the spatiotemporal feature output from several Transformer blocks processed in the current stage... Combination Implicit hints for time steps By prompting the Mamba module to update the spatiotemporal features, the updated spatiotemporal results are generated. With the updated implicit hints for:

[0061] (16)

[0062] In equation (16), Indicates the first The stage prompts the Mamba module; The values ​​can be 1, 2, 3, ..., N;

[0063] The time-space result after the Mamba module update will be displayed. With the template features Cascading, forming the cascading characteristics of the second stage. for:

[0064] (17)

[0065] Finally, the cascaded features of the second stage will be... As input to the second prompt encoder.

[0066] Furthermore, due to the excellent performance of the State-Space Model (SSM) in handling time series tasks, the Mamba module uses it as one of its core architectural components. Firstly, the State-Space Model (SSM) employs hidden state functions. This maps the input one-dimensional function x(t) to the output function y(t). , The calculation process is represented by a linear ordinary differential equation (ODE), and the calculation process is as follows:

[0067] (18)

[0068] (19)

[0069] In equations (18) and (19), These are all parameters of the State-Space Model (SSM). , , , Hidden state function ;

[0070] Then, in order to better adapt SSM to deep learning frameworks, the zero-order preservation (ZOH) method is used to adjust the parameters. and parameters Discretize:

[0071] (20)

[0072] (twenty one)

[0073] In equations (20) and (21), It is the identity matrix; Indicates the discretization time step; This represents the discretization result of parameter A; This represents the discretization result of parameter B;

[0074] The discretized state-space model (SSM) is as follows:

[0075] (twenty two)

[0076] (twenty three)

[0077] In equations (22) and (23), Let t be the hidden state at the t-th time step, which is the stored target historical state; This is the update result at time step t; The target state is the current input.

[0078] Finally, due to It has the ability to store and aggregate long-term time-series content, therefore it will This serves as an implicit cue for each stage, used in subsequent propagation. and localized enhanced spatiotemporal features The update is as follows:

[0079] (twenty four)

[0080] (25)

[0081] Further, step S4 includes the following steps:

[0082] S4-1. After several stages of content prompting encoder modeling, explicit prompts are implemented. Having aggregated temporal information about changes in the target's appearance, further optimization of the features of the current search region is needed. Explicit hints for calculating the final encoding are required. Features of the search area Similarity score between for:

[0083] (26)

[0084] In equation (26), the similarity score The higher the value, the greater the probability that the area belongs to the target;

[0085] S4-2, To

[0086] (27)

[0087] In formula (27), "

[0088] S4-3, Weighted search region features Input the tracking head to predict the target location.

[0089] Further, step S5 includes the following steps:

[0090] To effectively optimize the model, the overall model is constrained by classification loss and regression loss. The classification loss uses Focal Loss, and the regression loss integrates GIoU Loss and L1 Loss. To effectively balance the contributions of different loss terms during the optimization process, the model's total loss function is defined as follows:

[0091] (28)

[0092] In equation (28), Focal Loss() is the classification loss; Integrate GIoU Loss() for regression loss. Integrate L1 Loss() for regression loss; , Here, are the weighting coefficients,

[0093] The beneficial effects of this invention are as follows:

[0094] This invention achieves multiple gain effects in the field of visual object tracking through innovative network architecture and training strategies:

[0095] 1) Introducing Mamba into the visual backbone network fully utilizes Mamba's sequence modeling capabilities and constructs long-term representations at each stage, thereby enhancing the model's ability to capture target-related spatiotemporal dependencies. 2) The Mamba mechanism and the attention mechanism complement each other, which not only enhances the model's ability to capture the appearance of the target, but also effectively updates and stores the target-related representations through the hidden layer mechanism, thus effectively improving the model's ability to understand the target.

[0096] 3) The interaction of dual prompts provides the model with both global and local perspectives, further enhancing its ability to discriminate targets.

[0097] In summary, this invention has good generalization ability and robustness, can adapt to a variety of visual target tracking tasks, and shows strong adaptability to various challenges in different scenarios, providing an efficient and reliable solution for complex visual target tracking needs in fields such as autonomous driving and video security. Attached Figure Description

[0098] Figure 1 A comparative block diagram of existing technologies (a) and (b) in visual target tracking and the method described in the present invention (c);

[0099] Figure 2 This is a logical structure framework diagram of the present invention;

[0100] Figure 3 This is a schematic diagram of the Mamba module structure of the present invention;

[0101] Figure 4 This is a comparison chart of the metrics of this invention under various scenarios on the LaSOT dataset;

[0102] Figure 5 This is a visual comparison chart of the tracking results of this invention on the LaSOT dataset;

[0103] Figure 6 This is an attention heatmap of the temporal feature interest region in the LaSOT dataset according to the present invention. Detailed Implementation

[0104] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0105] In this embodiment, the Chinese meanings of the English letters are as follows:

[0106] Initial joint features;

[0107] To indicate the encoder input;

[0108] This indicates the superimposed output of multiple Transformer blocks in the encoder;

[0109] The result of attention calculation;

[0110] This is the residual result;

[0111] FFN stands for Feedforward Neural Network;

[0112] The time characteristics of each stage;

[0113] This is to indicate the decoupled template features in the encoder;

[0114] This is to indicate the features of the decoupled search region in the encoder;

[0115] The spatiotemporal features input to Mamba;

[0116] Locally enhanced spatiotemporal features in Mamba;

[0117] This is a phased implicit hint;

[0118] Implicit hints for updates – S3-5, Implicit hints for the current stage ;

[0119] SSM stands for State-Space Model;

[0120] This is the result of the right-hand branch;

[0121] This is the result of the left branch;

[0122] To reduce the dimensionality of the output;

[0123] To prompt for input from the Mamba module;

[0124] The results are from a secondary spatiotemporal modeling.

[0125] To indicate the spatiotemporal results after encoder update;

[0126] To provide an explicit hint;

[0127] The final encoding is displayed as a prompt;

[0128] The final encoded search region features;

[0129] Features of the weighted search region.

[0130] like Figure 2 The spatiotemporal single-target tracking method based on Mamba dual-cue, as shown, includes the following steps:

[0131] S1. First, several template images and a single search region image are acquired respectively. An image serialization processing strategy is used for progressive downsampling to obtain the feature sequences of the template images and the search region images respectively. A corresponding display prompt is added to each search region image. Then, all image feature sequences and display prompts are concatenated according to their combined width and height dimensions to form a joint feature. Step S1 includes the following steps:

[0132] S1-1. Acquire several template images Z and a single search region image X, respectively. , Processed according to the image serialization method Image patches of varying sizes are embedded, and then the original image is transformed through a two-stage linear layer merging process. Image patch embedding of size;

[0133] S1-2, will Image patch embedding of different sizes is reshaped along the spatial dimension to form a serialized template marker. and search area markers , , ;in, Indicates the number of input templates; Template tag The length of the marker, , The height of the template image. The width of the template image; Indicates the length of the marker in the search area. , The height of the search region image, The width of the search area image; D is the preset number of channels;

[0134] S1-3. First, mark each search area. Assign an additional learnable tag As a display prompt, t represents the time step of the current frame; then, in the template tag Search area markers With display prompts Learnable location embeddings are added to each element to enhance the corresponding spatial information; finally, display prompts with location embeddings are added. Template tags With search area markers Cascade along spatial dimensions to form initial joint features for:

[0135] (1)

[0136] In equation (1), L represents the length of the joint feature. D represents the preset number of channels; This indicates a cascading operation.

[0137] S2. Input the joint features obtained in step S1 into the first prompt encoder. The first prompt encoder performs spatiotemporal modeling on the joint features in several Transformer blocks. The spatiotemporal modeling includes appearance feature modeling and time feature modeling. The obtained spatiotemporal modeling results are separated into features of the search region and display time features, which are used as input to the prompt Mamba module. Step S2 includes the following steps:

[0138] S2-1. First, the Transformer-based backbone network is divided into N stages, and M Transformer blocks are assigned to each stage. The joint feature of each Transformer block is: Then, an independent prompting Mamba module is assigned after the Transformer block of each stage, while no prompting Mamba module is assigned after the Transformer block of the final stage, forming N prompting encoders.

[0139] S2-2, In the N cue encoders formed in step S2-1, the joint features in each Transformer block are... Perform linear projections separately to obtain... Matrix, computed using multi-head self-attention The matrix is:

[0140] (2)

[0141] (3)

[0142] In equations (2) and (3), The matrices represent the query matrix, the queried matrix, and the weight matrix, respectively. It is a linear projection function; For multi-head self-attention functions; The result of attention calculation;

[0143] S2-3, Apply the attention calculation results obtained in step S2-2 With joint features Perform residual joins to obtain residual results. for:

[0144] (4)

[0145] Then, the residual results The input is fed into a feedforward neural network for further processing, yielding the output of a single Transformer block. for:

[0146] (5)

[0147] In equations (4) and (5), For layer normalization processing; This is the residual result; The result of attention calculation; It is a feedforward neural network;

[0148] Finally, the Transformer blocks in the first prompt encoder are overlaid to form the Transformer block output. for:

[0149] (6)

[0150] In equation (6), For the Transformer block stacking operation in the first prompt encoder, M is the number of Transformer blocks allocated in each stage, and i takes the values ​​1, 2, 3, ..., M;

[0151] S2-4. First, the output of the Transformer block stacking operation. Decoupling was performed to obtain the time characteristics of the first stage. Template features and search area features for:

[0152] ; (7)

[0153] In equation (7), () indicates a decoupling operation;

[0154] Then, the time characteristics of the first stage and search area features By cascading, the spatiotemporal features related to the target can be obtained. Spatiotemporal characteristics As a prompt for the Mamba module input , The spatiotemporal features for:

[0155] ; (8)

[0156] In equation (8), Cascaded operations.

[0157] S3. First, the features and temporal features separated in step S2 are combined with staged implicit cues, and then the long-term spatiotemporal context information is aggregated in Mamba, and the features, temporal features, and implicit cues are updated. Then, the updated features and temporal features are residually connected with the input of Mamba to obtain the secondary spatiotemporal modeling result. Finally, the secondary spatiotemporal modeling result is used as the input of the second cue encoder for secondary spatiotemporal modeling. The cue Mamba module includes a left branch and a right branch. Step S3 is processed through two parallel branches, the left branch and the right branch, to simulate the long-term target dynamics, including the following steps:

[0158] S3-1, In the right-hand branch:

[0159] First, linear projection is used to extend the spatiotemporal features. Channel dimension :

[0160] (9)

[0161] In equation (9), It is a linear projection function;

[0162] Then, the channel dimensions are enhanced through convolution operations. The local spatial dependence is then activated using the SiLU activation function to generate locally enhanced spatiotemporal features. for:

[0163] (10)

[0164] In equation (10), Causal convolution function;

[0165] Finally, the spatiotemporal features of local enhancement will be used. Combined with from As input to the state-space model, the long-term context information is captured and updated through the state-space model to generate the right-hand branch result. And updated implicit hints for:

[0166] (11)

[0167] In equation (11), Calculations are performed for the state-space model;

[0168] S3-2, In the left branch:

[0169] First, a linear layer is used to expand the spatiotemporal features. Channel dimension :

[0170] (12)

[0171] In equation (12), It is a linear projection function;

[0172] Then, the SiLU activation function is used to activate the channel dimension. Generate left branch results for:

[0173] (13)

[0174] In equation (13),

[0175] S3-3, Results of the right-side branch and the results of the left branch The weighted sum is obtained by performing Hadamard multiplication operations together, and then the weighted result is reduced in dimensionality through linear projection to obtain the dimensionality-reduced output. for:

[0176] (14)

[0177] In equation (14), This is the result of the right-hand branch; This is the result of the left branch; Linear projection function; This represents the Hadamard multiplication operation;

[0178] S3-4. Dimensional Reduction Output Results Spatiotemporal characteristics of input to the Mamba module. The results of the secondary spatiotemporal modeling are as follows:

[0179] (15)

[0180] S3-5, Spatiotemporal feature output from several Transformer blocks processed in the current stage. Combination Implicit hints for time steps By prompting the Mamba module to update the spatiotemporal features, the updated spatiotemporal results are generated. With the updated implicit hints for:

[0181] (16)

[0182] In equation (16), Indicates the first The stage prompts the Mamba module; The values ​​can be 1, 2, 3, ..., N;

[0183] The time-space result after the Mamba module update will be displayed. With the template features Cascading, forming the cascading characteristics of the second stage. for:

[0184] (17)

[0185] Finally, the cascaded features of the second stage will be... As input to the second prompt encoder;

[0186] S3-6. First, the state-space model SSM adopts hidden state functions. This maps the input one-dimensional function x(t) to the output function y(t). , The calculation process is represented by a linear ordinary differential equation, and the calculation process is as follows:

[0187] (18)

[0188] (19)

[0189] In equations (18) and (19), These are all parameters of the State-Space Model (SSM). , , , Hidden state function ;

[0190] Then, the zero-order preserve method is used to adjust the parameters. and parameters Discretize:

[0191] (20)

[0192] (twenty one)

[0193] In equations (20) and (21), It is the identity matrix; Indicates the discretization time step; This represents the discretization result of parameter A; This represents the discretization result of parameter B;

[0194] The discretized state-space model (SSM) is as follows:

[0195] (twenty two)

[0196] (twenty three)

[0197] In equations (22) and (23), Let t be the hidden state at the t-th time step, which is the stored target historical state; This is the update result at time step t; The target state is the current input.

[0198] Finally, This serves as an implicit cue for each stage, used in subsequent propagation. and localized enhanced spatiotemporal features The update is as follows:

[0199] (twenty four)

[0200] (25)

[0201] S4. Repeat steps S2-S3, using the display prompt from the last prompt encoder output to weight the search area features, and input the weighted search area features into the tracking head to predict the target position; step S4 includes the following steps:

[0202] S4-1, Explicit hints for calculating the final code Features of the search area Similarity score between for:

[0203] (26)

[0204] In equation (26), the similarity score The higher the value, the greater the probability that the area belongs to the target;

[0205] S4-2, To

[0206] (27)

[0207] In formula (27), "

[0208] S4-3, Weighted search region features Input the tracking head to predict the target location.

[0209] S5. Optimize the model parameters using a joint loss function of Focal Loss, GIoU Loss, and L1 Loss to obtain the optimal model parameters for target tracking, thus completing spatiotemporal single-target tracking based on Mamba dual-cues; step S5 includes the following steps:

[0210] The model is constrained by classification loss and regression loss. The classification loss uses Focal Loss, and the regression loss integrates GIoU Loss and L1 Loss. To effectively balance the contributions of different loss terms during the optimization process, the overall loss function of the model is defined as follows:

[0211] (28)

[0212] In equation (28), Focal Loss() is the classification loss; Integrate GIoU Loss() for regression loss. Integrate L1 Loss() for regression loss; , Here, are the weighting coefficients,

[0213] Experimental verification was conducted on the spatiotemporal single-target tracking method based on Mamba dual-cue proposed in this invention. Training was performed using two NVIDIA RTX 3090 GPUs, while evaluation was conducted on a single NVIDIA RTX 3090 GPU. The model was optimized using AdamW during training, with the learning rate of the backbone network set to... And set the weight decay coefficient to The training process consists of 300 epochs, with 30,000 images used in each epoch, and the learning rate decays starting from the 240th epoch.

[0214] This embodiment follows mainstream tracking methods, using four datasets for training: LaSOT, COCO, Tracking Net, and GOT-10k. Evaluation is performed on the challenging LaSOT dataset, and the results are analyzed below:

[0215] like Figure 4 Figure 5 The paper presents both quantitative and qualitative comparisons of the present invention with other methods on the LaSOT dataset. Radar chart comparisons show that the present invention outperforms existing methods of similar scale in most challenging scenarios; its leading performance in target deformation and complete occlusion scenarios further demonstrates the effectiveness of the present invention. Visualization results show that the present invention exhibits significant advantages in real-world environments. Compared with other methods, the present invention maintains focus on the target throughout the video sequence without significant target shift, especially in scenarios with object deformation, occlusion, and motion blur, providing more accurate target location prediction.

[0216] like Figure 6 The heatmap of the search region that the temporal features of this invention focus on is shown. As the video progresses, the temporal features consistently focus on the key changing regions of the target, further demonstrating the stronger robustness of this invention in complex scenarios.

[0217] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A spatiotemporal single-target tracking method based on Mamba dual-cue, characterized in that, Includes the following steps: S1. First, collect several template images and a single search region image respectively. Use an image serialization processing strategy to perform progressive downsampling to obtain the feature sequences of the template images and the search region images respectively, and add corresponding display prompts for each search region image. Then, concatenate all image feature sequences and display prompts according to the merged width and height dimensions to form joint features. S2. Input the joint features obtained in step S1 into the first prompt encoder. Use the first prompt encoder to perform spatiotemporal modeling of the joint features in several Transformer blocks. The spatiotemporal modeling includes appearance feature modeling and time feature modeling. Separate the obtained spatiotemporal modeling results into features of the search area and display time features, and use them as input to the prompt Mamba module. S3. First, the features and temporal features separated in step S2 are combined with the staged implicit cues, and then the long-term spatiotemporal context information is aggregated in Mamba, and the features, temporal features and implicit cues are updated. Then, the updated features and temporal features are residually connected with the input of Mamba to obtain the secondary spatiotemporal modeling result. Finally, the secondary spatiotemporal modeling result is used as the input of the second cue encoder to perform secondary spatiotemporal modeling. S4. Repeat steps S2~S3, use the display prompts output by the last prompt encoder to weight the search area features, input the weighted search area features into the tracking head, and predict the target position. S5. The model parameters are optimized by using a joint loss function of Focal Loss, GIoU Loss and L1 Loss to obtain the optimal model parameters for target tracking, thus completing spatiotemporal single target tracking based on Mamba dual cueing.

2. The spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 1, characterized in that, Step S1 includes the following steps: S1-1. Acquire several template images Z and a single search region image X, respectively. , Processed according to the image serialization method Image patches of varying sizes are embedded, and then the original image is transformed through a two-stage linear layer merging process. Image patch embedding of size; S1-2, will Image patch embedding of different sizes is reshaped along the spatial dimension to form a serialized template marker. and search area markers , , ;in, Indicates the number of input templates; Template tag The length of the marker, , The height of the template image. The width of the template image; Indicates the length of the marker in the search area. , The height of the search region image, The width of the search area image; D is the preset number of channels; S1-3. First, mark each search area. Assign an additional learnable tag As a display prompt, t represents the time step of the current frame; then, in the template markup Search area markers With display prompts Learnable location embeddings are added to each element to enhance the corresponding spatial information; finally, display prompts with location embeddings are added. Template tags With search area markers Cascade along spatial dimensions to form initial joint features for: （1） In equation (1), L represents the length of the joint feature. D represents the preset number of channels; This indicates a cascading operation.

3. The spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 1, characterized in that, Step S2 includes the following steps: S2-1. First, the Transformer-based backbone network is divided into N stages, and M Transformer blocks are assigned to each stage. The joint feature of each Transformer block is: Then, an independent prompting Mamba module is assigned after the Transformer block of each stage, while no prompting Mamba module is assigned after the Transformer block of the final stage, forming N prompting encoders. S2-2, In the N cue encoders formed in step S2-1, the joint features in each Transformer block are... Perform linear projections separately to obtain... Matrix, computed using multi-head self-attention The matrix is: （2） （3） In equations (2) and (3), The matrices represent the query matrix, the queried matrix, and the weight matrix, respectively. It is a linear projection function; For multi-head self-attention functions; The result of attention calculation; S2-3, Apply the attention calculation results obtained in step S2-2 With joint features Perform residual joins to obtain residual results. for: （4） Then, the residual results The input is fed into a feedforward neural network for further processing, yielding the output of a single Transformer block. for: ； （5） In equations (4) and (5), For layer normalization processing; For residual results; The result of attention calculation; It is a feedforward neural network; Finally, the Transformer blocks in the first prompt encoder are overlaid to form the Transformer block output. for: （6） In equation (6), For the Transformer block stacking operation in the first prompt encoder, M is the number of Transformer blocks allocated in each stage, and i takes the values ​​1, 2, 3, ..., M; S2-4. First, the output of the Transformer block stacking operation. Decoupling was performed to obtain the time characteristics of the first stage. Template features and search area features for: ； （7） In equation (7), () indicates a decoupling operation; Then, the time characteristics of the first stage and search area features By cascading, the spatiotemporal features related to the target can be obtained. Spatiotemporal characteristics As a prompt for the Mamba module input , The spatiotemporal features for: ； （8） In equation (8), This indicates a cascading operation.

4. A spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 1 or 3, characterized in that, The Mamba module mentioned includes a left branch and a right branch. Step S3 processes the data through two parallel branches, the left branch and the right branch, to simulate long-term target dynamics, including the following steps: S3-1, In the right-hand branch: First, linear projection is used to extend the spatiotemporal features. Channel dimension : （9） In equation (9), It is a linear projection function; Then, the channel dimensions are enhanced through convolution operations. The local spatial dependence is then activated using the SiLU activation function to generate locally enhanced spatiotemporal features. for: （10） In equation (10), For activation functions; for Causal convolution function; It is a linear projection function; Finally, the spatiotemporal features of local enhancement will be used. Combined with from Implicit hints for time steps As input to the state-space model, the long-term context information is captured and updated through the state-space model to generate the right-hand branch result. And updated implicit hints for: ； （11） In equation (11), Calculations are performed for the state-space model; S3-2, In the left branch: First, a linear layer is used to expand the spatiotemporal features. Channel dimension : ； （12） In equation (12), It is a linear projection function; Then, the SiLU activation function is used to activate the channel dimension. Generate the left branch result for: ； （13） In equation (13), For activation functions; S3-3, Results of the right-side branch and the results of the left branch The weighted sum is obtained by performing Hadamard multiplication operations together, and then the weighted result is reduced in dimensionality through linear projection to obtain the dimensionality-reduced output. for: （14） In equation (14), This is the result of the right-hand branch; This is the result of the left branch; It is a linear projection function; "Indicates the Hadamard multiplication operation; S3-4. Dimensional Reduction Output Results Spatiotemporal characteristics of input to the Mamba module. Performing residual connections yields the following quadratic spatiotemporal modeling results: （15） S3-5, Spatiotemporal feature output from several Transformer blocks processed in the current stage. Combination Implicit hints for time steps By prompting the Mamba module to update the spatiotemporal features, the updated spatiotemporal results are generated. With the updated implicit hints for: （16） In equation (16), Indicates the first The Mamba module provides stage-specific hints; The values ​​can be 1, 2, 3, ..., N; The time-space result after the Mamba module update will be displayed. With the template features Cascading, forming the cascading characteristics of the second stage. for: （17） Finally, the cascaded features of the second stage will be... As input to the second prompt encoder.

5. The spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 4, characterized in that, First, the state-space model SSM uses hidden state functions. , The input is a one-dimensional function. Mapping to output function , , The calculation process is represented by a linear ordinary differential equation, and the calculation process is as follows: （18） （19） In equations (18) and (19), These are all parameters of the State-Space Model (SSM). , , , ; Hidden state function The first derivative; Then, the zero-order preserve method is used to adjust the parameters. and parameters Discretize: （20） （21） In equations (20) and (21), It is the identity matrix; Indicates the discretization time step; This represents the discretization result of parameter A; This represents the discretization result of parameter B; The discretized state-space model (SSM) is as follows: （22） （23） In equations (22) and (23), Let t be the hidden state at the t-th time step, i.e. the stored target historical state; This represents the update result at time step t. The target state is the current input. Finally, This serves as an implicit cue for each stage, used in subsequent propagation. Implicit hints for staged input and localized enhanced spatiotemporal features The update is as follows: （24） （25）。 6. The spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 1, characterized in that, Step S4 includes the following steps: S4-1, Explicit hints for calculating the final code Features of the search area Similarity score between for: （26） In equation (26), the similarity score The higher the value, the greater the probability that the area belongs to the target; S4-2, To Element-wise weighting is performed to enhance the feature representation of the target region while suppressing background interference. ； （27） In equation (27), " " indicates element-wise multiplication; S4-3, Weighted search region features Input the tracking head to predict the target location.

7. The spatiotemporal single-target tracking method based on Mamba dual-cue as described in claim 1, characterized in that, Step S5 includes the following steps: The model is constrained by classification loss and regression loss. The classification loss uses Focal Loss, and the regression loss integrates GIoU Loss and L1 Loss. To effectively balance the contributions of different loss terms during the optimization process, the overall loss function of the model is defined as follows: （28） In equation (28), Focal Loss() is the classification loss; Integrate GIoU Loss() for regression loss. Integrate L1 Loss() for regression loss; , , Here, are the weighting coefficients, ,

Images