A visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of state-space model

Abstract

The present invention provides a visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model, which belongs to the fields of machine learning, computer vision, and target tracking. By utilizing the linear long sequence modeling capability of the visual state space model, long-term cross-frame integration components and short-term historical trajectory cues are embedded into a bidirectional visual state space model encoder (Mamba), and unified spatiotemporal context modeling of global appearance information and local motion information is performed in a single-stream framework. The present invention breaks away from the limitation that the computational complexity of the attention mechanism grows with the square of the sequence length, and for the first time proposes a visible light-thermal infrared tracking framework based on state space models (State Space Models). The method of the present invention utilizes the advantage that its computational complexity grows linearly with the sequence length, and can achieve efficient operation under multi-source long sequence input with lower computational complexity.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, computer vision, and target tracking, and relates to a visual state space algorithm, a long-term appearance and short-term trajectory cue learning algorithm, and a single-stream network model; specifically, it is a visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model. Background Art

[0002] Given a video sequence and the target box annotation of a specific target in the initial frame, the visible light-thermal infrared target tracking algorithm aims to estimate the position state of the specific target in subsequent video frames using visible light and thermal infrared modal images. Compared with tracking based on pure visible light, thermal infrared images can perceive targets with different thermal radiation, especially in extreme lighting and low-texture environments, and can provide additional target information for visible light images, enhancing the robustness and adaptability of tracking algorithms under all-weather conditions. Therefore, in recent years, the field of visible light-thermal infrared target tracking has focused more on the research of various multimodal feature extraction and information interaction fusion modules and technologies, further advancing the performance of visible light-thermal infrared target tracking algorithms, and laying the foundation for the practical application of visual tracking in many fields such as national defense, medical care, and transportation.

[0003] With the development of deep learning technology, visible light-thermal infrared target tracking algorithms have gradually been introduced from traditional convolutional neural network methods to methods based on Transformer self-attention mechanism networks, including dual-stream methods and single-stream methods. Dual-stream tracking algorithms (mfDiMP, CAT++, TBSI, etc.) usually use a parameter-sharing visual Transformer backbone network for visible light modality and thermal infrared modality to perform modality-related feature extraction and top-level feature fusion modules to achieve the fusion of extracted visible light features and thermal infrared features. This type of method conforms to the intuitive design idea of ​​early heterogeneous modality information extraction and fusion and the limitation that convolutional neural networks can only accept fixed-length data, and has strong interpretability. However, due to the need to perform forward reasoning of the backbone network twice, and the heterogeneous modality features are extracted and then interacted separately, the tracking speed is slow, and the target complementary information interaction is insufficient. Single-stream tracking methods (ViPT, USTrack, UnTrack, etc.) unify the extraction and interaction of different modal information into a compact Transformer network. This not only improves the speed of the tracking algorithm, but also ensures the full complementary utilization of different modal information, achieving advanced tracking performance.

[0004] The above tracking algorithms all adopt the idea of ​​image-level appearance matching. Although the advanced single-stream network further improves the ability to distinguish targets, it still lacks the ability to deal with the deformation and appearance changes of targets during the tracking process, and targets with similar appearance. In order to deal with the problem of appearance changes during the tracking process, Hongyu Wang et al. proposed a template online update method TATrack in the paper "TemporalAdaptive RGBT Tracking with Modality Prompt". This method uses dynamic templates to capture high-quality templates in online tracking as a supplement to the initial template to enrich the reference information of the tracked target. However, this method only uses a single-frame dynamic template, lacks the ability to model long-term spatiotemporal context sequence, and becomes a two-stream network, which increases the computational burden.

[0005] Therefore, how to design an efficient single-stream multimodal tracking model for unified feature extraction and interaction, and capture the real long-term and short-term temporal context to deal with the temporal changes in target appearance and the problem of similar targets is one of the challenges that need to be urgently solved in the current visible light-thermal infrared target tracking task. Summary of the invention

[0006] The present invention aims to provide a visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of state space models. By utilizing the linear long sequence modeling capability of the visual state space model, the long-term cross-frame integration components and short-term historical trajectory cues are embedded into the bidirectional visual state space model encoder (Mamba), and the spatiotemporal context of global appearance information and local motion information is uniformly modeled in a single-stream framework. The present invention breaks away from the limitation that the computational complexity of the attention mechanism grows with the square of the sequence length, and for the first time proposes a visible light-thermal infrared tracking framework based on state space models. The method of the present invention takes advantage of its computational complexity growing linearly with the sequence length, and can achieve efficient operation under multi-source long sequence input with low computational complexity.

[0007] The technical solution of the present invention:

[0008] A visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model, the steps are as follows:

[0009] Step 1: In the first stage of model training, the long-term cross-frame integration component is embedded into the bidirectional visual Mamba encoder, and the video-level input sequence is used to train the single-stream bidirectional visual Mamba encoder to learn the ability to integrate long-term cross-frame appearance information. The specific process is as follows:

[0010] Step 1.1: Randomly extract from the same video sequence within the maximum sampling interval Frames are used as template image pairs, and 1 frame is extracted as search image pairs. Both template image pairs and search image pairs contain visible light and thermal infrared dual-modal images. Represent the template image pairs of visible light and thermal infrared, respectively, and use Represent the search image pairs of visible light and thermal infrared single frames respectively, where are the height and width of the template image pair containing the target at the center after the template frame is cropped and scaled, are the height and width of the search image pair containing the target at the center after the search frame has been cropped and scaled.

[0011] Step 1.2: Before being input into the bidirectional visual Mamba encoder, all template image pairs and search image pairs are passed through an image block embedding layer to embed all original 3D images into the feature space to become embedded template vectors. and embedding search vector In the embedding template vector , in the embedding search vector Add positional encoding element by element ,in, , Represents the length of the template vector and the search vector, is the dimension of the embedding space.

[0012] Step 1.3: Concatenate all the embedded template vectors and embedded search vectors with positional encoding added along the length dimension and pass them through the stacked Layer Bidirectional Visual Mamba Encoder :

[0013] V ts 0 =[ V t RGB ; V s RGB ; V t TIR ; V s TIR ]

[0014]

[0015] The bidirectional visual Mamba encoder It mainly includes a normalization layer, a linear input mapping layer, a selective structured state space scanning module S6, and a linear output mapping layer.

[0016] (A) Input image patch embedding vector First, it is normalized by a normalization layer and then processed by two linear input mapping layers to convert the dimensions Ascending to the inner dimension The intermediate features obtained from this process are expressed as , this process is expressed mathematically as:

[0017]

[0018]

[0019] Among them, RN and They represent RMS Normalization and dimension-increasing linear layer respectively.

[0020] (B) Intermediate features It will pass through two branches. After activation by SiLU activation function, it is similar to that after one-dimensional convolution, SiLU activation function and selective structured state space scanning module S6. Perform Hadamard product. Intermediate features It will go through a forward and reverse bidirectional scan.

[0021] (C) Intermediate features of forward and reverse bidirectional scanning After element-by-element addition, the inner dimensions are transformed through a linear output mapping layer. Dimensionality reduction to dimension ;

[0022] (D) Repeat steps (A) to (C) in this way, and pass through the stacked The whole process is stacked Layer Bidirectional Visual Mamba Encoder The process is expressed mathematically as follows, Represents two scanning directions: forward and backward:

[0023]

[0024] in, It stands for the selective structured state-space scanning module.

[0025] Further: The difference from the general structured state space scanning method is that its state space parameters are linearly time-varying, which means that the weight of the state space is dynamically adjusted according to the input data, the parameters change over time and the model selectively selects the optimal parameters. This makes the model context- and semantic-aware and can better model sequence data. Specifically, the state space-based model is derived from a continuous system that is derived from a hidden state. The one-dimensional sequence From input Mapping to output . Using the evolution parameter and mapping parameters and The sequence-to-sequence transformation is defined as follows:

[0026]

[0027]

[0028] In order to process digital signals, the above continuous system needs to be discretized, and the evolution parameters, mapping parameters and input hidden states need to be discretized:

[0029]

[0030]

[0031]

[0032]

[0033] in is the time scale parameter, and Represents the discrete hidden states at different time steps. Parameters and is a parameter and The corresponding discrete form of .

[0034] Step 2: Based on the first stage training, embed the short-term history trajectory cues into the bidirectional visual Mamba encoder and add the second stage training. The second stage training process is as follows:

[0035] Step 2.1: Convert continuous coordinates into a sequence of discrete vectors using a shared vocabulary. Specifically:

[0036] First, using the model trained in the first stage, we re-infer each video sequence in the training set to obtain pseudo labels on the training set. Pseudo labels are target coordinates predicted by the inaccurate model.

[0037] Next, let the index of the current search area frame be , before sampling Pseudo-labels of frames (a local neighborhood) as short-term history trajectory hints:

[0038] { C t-j = [ x min t-j , y min t-j , x max t-j , y max t-j ]∣j∈{1,2,…, T}}

[0039] in, Indicates The horizontal axis coordinate of the upper left corner of the frame target, Indicates The vertical coordinate of the upper left corner of the frame target, Indicates The horizontal axis coordinate of the lower right corner of the frame target, Indicates The vertical coordinate of the lower right corner of the frame target, Indicates The frame target position state consists of four consecutive coordinate values;

[0040] The target position status of different frames , , are mapped to the same current search area coordinate system, and the historical trajectory prompt set obtained after mapping is recorded as .

[0041] Finally, each The continuous coordinate values ​​in will be uniformly normalized and discretized into integers between [1, nbins]. Each integer acts as an index to retrieve the coordinate embedding vector from the shared vocabulary .

[0042] Step 2.2: Use the expansion coefficient to expand the actual representation range of the shared vocabulary. In addition, as the coordinate embedding vector Add a coordinate query vector , used to calculate the L1 loss between the true value and the true value for auxiliary supervision.

[0043] Step 2.3: Concatenate the image patch embedding vector, coordinate embedding vector, and coordinate query vector and feed them into the bidirectional visual Mamba encoder for unified context modeling. Now the input and output of each layer of the bidirectional visual Mamba encoder are restated as follows:

[0044] .

[0045] Step 3: In the tracking model inference phase, an online template memory storage selection mechanism is designed to store and filter the multi-frame templates in the tracking process. The following formula is used in the online template memory storage selection mechanism to determine which frames are selected as memory template frames:

[0046]

[0047] in represents the average memory duration of each template. Indicates the index of the current frame.

[0048] In addition, a historical position queue needs to be maintained during the inference phase to provide local target trajectory information. Whenever the predicted position is output for the current frame, the historical position queue is updated with the current coordinates in a first-in, first-out manner.

[0049] Beneficial effects of the present invention:

[0050] (1) This paper proposes for the first time to use the linear long sequence modeling advantages of Mamba to explore the spatiotemporal context modeling of visible light-thermal infrared tracking, providing the community with a simple and effective pure Mamba baseline, which is conducive to the subsequent research of more visible light-thermal infrared tracking algorithms based on Mamba.

[0051] (2) The present invention integrates the global long-term cross-frame appearance and the local short-term historical trajectory hints in a compact single-stream framework, improving the ability of the visible light-thermal infrared tracking algorithm to cope with complex tracking scenarios such as target appearance changes and similar target occlusion interference, and improving tracking robustness and accuracy. It can also run in real time, reducing computational complexity and system memory requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the overall structural framework diagram of visible light-thermal infrared tracking based on long-term and short-term spatiotemporal context modeling.

[0053] Figure 2 This is the structure diagram of the bidirectional visual Mamba encoder. DETAILED DESCRIPTION

[0054] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0055] A visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model, the steps are as follows:

[0056] Step 1: In the first stage of model training, the long-term cross-frame integration component is embedded into the bidirectional visual Mamba encoder, and the video-level input sequence is used to train the single-stream bidirectional visual Mamba encoder to learn the ability to integrate long-term cross-frame appearance information. The specific process is as follows:

[0057] Step 1.1: Randomly extract from the same video sequence within the maximum sampling interval Frames are used as template image pairs, and 1 frame is extracted as search image pairs. Both template image pairs and search image pairs contain visible light and thermal infrared dual-modal images. Represent the template image pairs of visible light and thermal infrared, respectively, and use Represent the search image pairs of visible light and thermal infrared single frames respectively, where are the height and width of the template image pair containing the target at the center after the template frame is cropped and scaled, are the height and width of the search image pair containing the target at the center after the search frame is cropped and scaled. This method samples frames at different times as video-level training samples, which has the advantage of spatiotemporal context modeling compared to the original method's image-pair-level training samples.

[0058] Step 1.2: Before being input into the bidirectional visual Mamba encoder, all template image pairs and search image pairs are passed through an image block embedding layer to embed all original 3D images into the feature space to become embedded template vectors. and embedding search vector In order to provide the model with the ability to perceive the spatial position of the image block, the template vector is embedded Add positional encoding element by element and, embedding search vector Add positional encoding element by element ,in, , Represents the length of the template vector and the search vector, is the dimension of the embedding space.

[0059] Step 1.3: Concatenate all the embedded template vectors and embedded search vectors with positional encoding added along the length dimension and pass them through the stacked Layer Bidirectional Visual Mamba Encoder :

[0060] V ts 0 =[ V t RGB ; V s RGB ; V t TIR ; V s TIR ]

[0061]

[0062] The bidirectional visual Mamba encoder It mainly includes a normalization layer, a linear input mapping layer, a selective structured state space scanning module S6, and a linear output mapping layer.

[0063] (A) Input image patch embedding vector First, it is normalized by a normalization layer and then processed by two linear input mapping layers to convert the dimensions Ascending to the inner dimension The intermediate features obtained from this process are expressed as , this process is expressed mathematically as:

[0064]

[0065]

[0066] Among them, RN and They represent RMS Normalization and dimension-increasing linear layer respectively.

[0067] (B) Intermediate features It will pass through two branches. After activation by SiLU activation function, it is similar to that after one-dimensional convolution, SiLU activation function and selective structured state space scanning module S6. In order to make the elements in the sequence have a global receptive field and allow the sequence elements to interact with each other without restriction, the intermediate features It will go through a forward and reverse bidirectional scan.

[0068] (C) Intermediate features of forward and reverse bidirectional scanning After element-by-element addition, the inner dimensions are transformed through a linear output mapping layer. Dimensionality reduction to dimension ;

[0069] (D) Repeat steps (A) to (C) in this way, and pass through the stacked The whole process is stacked Layer Bidirectional Visual Mamba Encoder The process is expressed mathematically as follows, Represents two scanning directions: forward and reverse:

[0070]

[0071] in, represents the selective structured state space scanning module. Further: The difference from the general structured state space scanning method is that its state space parameters are linearly time-varying, which means that the weight of the state space is dynamically adjusted according to the input data, the parameters change over time and the model selectively selects the optimal parameters. This makes the model context- and semantic-aware and can better model sequence data. Specifically, the state space-based model is derived from a continuous system that is derived from a hidden state. The one-dimensional sequence From input Mapping to output . Using the evolution parameter and mapping parameters and The sequence-to-sequence transformation is defined as follows:

[0072]

[0073]

[0074] In order to process digital signals, the above continuous system needs to be discretized. Modern state space models, such as Mamba, usually use the zero-order hold (ZOH) method to complete it. It is necessary to discretize the evolution parameters, mapping parameters and input hidden states:

[0075]

[0076]

[0077]

[0078]

[0079] in is the time scale parameter, and Represents the discrete hidden states at different time steps. Parameters and is a parameter and The corresponding discrete form of .

[0080] Step 2: Although multi-frame input provides the model with a global perspective across the video to understand the tracked target, the model still lacks robustness when facing moving targets that are affected by similar backgrounds or occluded. Therefore, in order to supplement the model's single appearance perception ability, short-term historical trajectory cues are embedded in the bidirectional visual Mamba encoder. On the basis of the first stage training, a second stage of training is added. This stage incorporates a mechanism for using short-term historical trajectory cues, which prompts the model to learn to reason based on short-term local historical motion information, improving the model's tracking performance in complex environments, especially when appearance information alone cannot effectively distinguish foreground targets. The training process of the second stage is as follows:

[0081] Step 2.1: Convert continuous coordinates into a sequence of discrete vectors using a shared vocabulary. Specifically:

[0082] First, using the model trained in the first stage, we re-infer each video sequence in the training set to obtain pseudo labels on the training set. Pseudo labels are target coordinates predicted by the inaccurate model.

[0083] Next, let the index of the current search area frame be t, and the continuous Pseudo-labels of frames (a local neighborhood) as short-term history trajectory hints:

[0084] { C t-j = [ x min t-j , y min t-j , x max t-j , y max t-j ]∣j∈{1,2,…, T}}

[0085] in, Indicates The horizontal axis coordinate of the upper left corner of the frame target, Indicates The vertical coordinate of the upper left corner of the frame target, Indicates The horizontal axis coordinate of the lower right corner of the frame target, Indicates The vertical coordinate of the lower right corner of the frame target, Indicates Frame target position state, which consists of four consecutive coordinate values.

[0086] The target position status of different frames , , are mapped to the same current search area coordinate system, and the historical trajectory prompt set obtained after mapping is recorded as .

[0087] Finally, each The continuous coordinate values ​​in will be discretized uniformly into integers between [1, nbins]. Each integer acts as an index to retrieve a single coordinate embedding vector from the shared vocabulary , so the target position state of each frame Convert to a discrete coordinate embedding vector of length 4 .

[0088] Step 2.2: Use the expansion coefficient To expand the actual representation range of the shared vocabulary, so as to effectively include trajectory coordinate information from further back. In addition, in order to force the model to learn to use trajectory information for positioning during training, we embed the vector Add a coordinate query vector , used to calculate the L1 loss between the true value and the true value for auxiliary supervision.

[0089] Step 2.3: Concatenate the image patch embedding vector, coordinate embedding vector, and coordinate query vector and feed them into the bidirectional visual Mamba encoder for unified context modeling. Now the input and output of each layer of the bidirectional visual Mamba encoder are restated as follows:

[0090] [ V ts l+1 ; V coord l+1 ; V query l+1 ]= E l + 1 ([ V ts l ; V coord l ; V query l ])

[0091] So far, the holistic tracking framework adds a local motion-based reasoning perspective to complement the global appearance-based reasoning for robust tracking. The linear growth of the computational complexity of the Mamba architecture with the sequence length ensures that the computational cost is affordable even when the input includes multi-modal and multi-source input data.

[0092] Step 3: In the tracking model inference phase, an online template memory storage selection mechanism is designed to store and filter the multi-frame templates in the tracking process, providing the model with rich and comprehensive video-level global target appearance information and maintaining the consistency of the training and testing process. The online template memory storage selection mechanism uses the following formula to determine which frames are selected as memory template frames:

[0093]

[0094] in represents the average memory duration of each template. Represents the index of the current frame. This sampling method not only considers the initial frame template that contains the most accurate target appearance, but also includes targets from later tracking stages close to the current frame as dynamic templates, which enables the model to exploit spatiotemporal context information across the entire tracking video to adapt to changes in template appearance during tracking.

[0095] In addition, a historical position queue needs to be maintained during the inference phase to provide local target trajectory information. Whenever the predicted position is output for the current frame, the historical position queue is updated with the current coordinates in a first-in, first-out manner.

[0096] Example:

[0097] (1) Initialize and pre-train a bidirectional Mamba single-stream network using ImageNet parameters.

[0098] In order to make a fair comparison with previous Transformer model-based methods and ensure the final results, we continued the paradigm of using the model parameters under classification data as initialization parameters and using conventional large-scale visible light datasets for pre-training. Specifically, after comparison, our method uses the model parameters that have been trained on ImageNet classification data in the VideoMamba work as the initialization parameters of our tracker, and uses four conventional visible light datasets, COCO, LaSOT, GOT10k and TrackingNet, to pre-train the tracking data for these parameters. The model parameters generated after pre-training are used as the initial parameters for our training in the visible light-thermal infrared field.

[0099] Table 1 shows the effects of different initialization parameters and pre-training strategies on the results. Based on the accuracy and success rate results, it is not difficult to find that pre-training is crucial for performance assurance, whether using ImageNet or visible light tracking data. And VideoMamba (which has the same architecture as Vim) pre-training brings better tracking performance than Vision Mamba (Vim) pre-training, indicating better and stronger feature representation. Large-scale pre-training using visible light can provide a good feature expression foundation for subsequent visible light-thermal infrared training.

[0100] Table 1: Effects of different initialization parameters and pre-training strategies on the results

[0101]

[0102] (2) Build and implement the proposed long- and short-term spatiotemporal context modeling.

[0103] Figure 1 Shown is the overall framework of the long-term and short-term spatiotemporal context modeling proposed by the present invention. The training input of the first stage includes 7 frames of visible light-thermal infrared template image pairs and 1 frame of visible light-thermal infrared search image pairs. These image inputs constitute long-term cross-frame appearance modeling, which is conducive to the model learning template priors in different time and space. In addition to the above-mentioned image inputs, the second stage of training adds 7 frames of historical position coordinates as a supplement to the running information. The historical position coordinates include the horizontal and vertical positions of the upper left corner and the lower right corner. Such continuous coordinates constitute a short-term local trajectory. When there is similar background interference or the target is occluded, the model can still infer the approximate direction of the current target after being occluded based on the previous position, so as not to drift to other abnormally displaced interference objects, which helps to improve the robustness of the model and the re-retrieval after occlusion.

[0104] Figure 2 The following is a diagram of the bidirectional visual Mamba encoder architecture. The linear input mapping layer is transformed from the dimension Upgrading to dimension The subsequent one-dimensional convolution is performed in the forward and reverse directions respectively. After the SiLU activation function, it is sent to the forward and reverse selective scanning algorithms respectively, and is multiplied element by element with the residual information of the other path respectively. Finally, the sum is added and sent to the linear output mapping layer. Dimensionality reduction to dimension After outputting from the stacked bidirectional visual Mamba encoder, the head prediction network reshapes the current frame search feature area and inputs it into the center position prediction branch, center position offset prediction branch, and rectangular frame width and height prediction branch respectively to predict the target frame state.

[0105] After offline training, the trained model will be used to continuously track a sequence in the inference stage. During the inference process, the present invention constructs an online template memory storage selection mechanism to store the initial frame search area features and the intermediate frame features across the entire video sampling during the tracking process to preserve rich target appearance information. In addition, a historical trajectory queue is constructed to suppress or punish interferers that are not in the trajectory prediction path through historical trajectories, so that the tracking process is stable and will not jump significantly.

[0106] Table 2: Effects of different initialization parameters and pre-training strategies on the results

[0107]

[0108] Table 2 shows the performance comparison between the algorithm of the present invention and the advanced algorithm, as well as the performance improvement results brought about by adding short-term historical trajectory modeling. The results show that the performance brought about by long-range appearance modeling has exceeded the level of the previous advanced algorithm, and the amount of training parameters has been greatly reduced, thereby reducing the system memory requirements and training costs. The addition of trajectory prompts has improved the ability of the tracking algorithm of the present invention to cope with complex tracking scenarios such as similar target occlusion interference, improved tracking accuracy and success rate, and the speed meets the real-time operation requirements.

Claims

1. A visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model, characterized in that: Here are the steps: Step 1: In the first stage of model training, the long-term cross-frame integration component is embedded into the bidirectional visual Mamba encoder, and the video-level input sequence is used to train the single-stream bidirectional visual Mamba encoder to learn the ability to integrate long-term cross-frame appearance information; Step 2: Based on the first stage training, embed the short-term historical trajectory cues into the bidirectional visual Mamba encoder and add the second stage training; Step 3: In the tracking model inference phase, an online template memory storage selection mechanism is designed to store and filter multi-frame templates during the tracking process; The specific process of step 1 is as follows: Step 1.1: Randomly extract from the same video sequence within the maximum sampling interval Frames are used as template image pairs, and one frame is extracted as search image pairs. Both template image pairs and search image pairs contain visible light and thermal infrared dual-modal images. Represent the template image pairs of visible light and thermal infrared, respectively, and use Represent the search image pairs of visible light and thermal infrared single frames respectively, where are the height and width of the template image pair containing the target at the center after the template frame is cropped and scaled, are the height and width of the search image pair containing the target at the center after the search frame has been cropped and scaled; Step 1.2: Before being input into the bidirectional visual Mamba encoder, all template image pairs and search image pairs are passed through an image block embedding layer to embed all original 3D images into the feature space to become embedded template vectors. and embedding search vector ; In the embedding template vector , in the embedding search vector Add positional encoding element by element ,in, , represents the length of the template vector and the search vector, is the dimension of the embedding space; Step 1.3: Concatenate all the embedded template vectors and embedded search vectors with positional encoding added along the length dimension and pass them through the stacked Layer Bidirectional Visual Mamba Encoder : The bidirectional visual Mamba encoder It mainly includes normalization layer, linear input mapping layer, selective structured state space scanning module S6, and linear output mapping layer; (A) Input image patch embedding vector First, it is normalized by a normalization layer and then processed by two linear input mapping layers to convert the dimensions Ascending to the inner dimension ; The intermediate features obtained from this process are expressed as , this process is expressed mathematically as: Among them, RN and They represent RMS Normalization and dimension-increasing linear layer respectively; (B) Intermediate features It will pass through two branches. After activation by SiLU activation function, it is similar to that after one-dimensional convolution, SiLU activation function and selective structured state space scanning module S6. Perform Hadamard product; intermediate features It will go through a forward and reverse bidirectional scan; (C) Intermediate features of forward and reverse bidirectional scanning After element-by-element addition, the inner dimensions are transformed through a linear output mapping layer. Dimensionality reduction to dimension ; (D) Repeat steps (A) to (C) in this way, and pass through the stacked Layer bidirectional visual Mamba encoder; the entire stacked Layer Bidirectional Visual Mamba Encoder The process is expressed mathematically as follows, Represents two scanning directions: forward and reverse: in, It represents the selective structured state space scanning module; The training process of the second stage is as follows: Step 2.1: Convert continuous coordinates to a sequence of discrete vectors using a shared vocabulary; specifically: First, use the model trained in the first stage to re-infer each video sequence in the training set to obtain pseudo labels on the training set. Pseudo labels are the target coordinates predicted by the inaccurate model. Next, let the index of the current search area frame be , before sampling Pseudo-labels of frames as short-term history trajectory hints: in, Indicates The horizontal axis coordinate of the upper left corner of the frame target, Indicates The vertical coordinate of the upper left corner of the frame target, Indicates The horizontal axis coordinate of the lower right corner of the frame target, Indicates The vertical coordinate of the lower right corner of the frame target, Indicates The frame target position state consists of four consecutive coordinate values; The target position status of different frames , , are mapped to the same current search area coordinate system, and the historical trajectory prompt set obtained after mapping is recorded as ; Finally, each The continuous coordinate values ​​in will be uniformly normalized and discretized into integers between [1, nbins]; each integer acts as an index to retrieve the coordinate embedding vector from the shared vocabulary ; Step 2.2: Use the expansion coefficient to expand the actual representation range of the shared vocabulary; in addition, as the coordinate embedding vector Add a coordinate query vector , used to calculate L1 loss between the true value and the true value for auxiliary supervision; Step 2.3: Concatenate the image patch embedding vector, coordinate embedding vector, and coordinate query vector and feed them into the bidirectional visual Mamba encoder for unified context modeling; now the input and output of each layer of the bidirectional visual Mamba encoder are restated as follows: 。 2. According to claim 1, a visible light-thermal infrared target tracking method based on robust spatiotemporal context modeling of a state space model is characterized in that: In step 3, the following formula is used in the online template memory storage selection mechanism to determine which frames are selected as memory template frames: in represents the average memory duration of each template; Indicates the index of the current frame; In addition, a historical position queue needs to be maintained during the inference phase to provide local target trajectory information. Whenever the predicted position is output for the current frame, the historical position queue is updated with the current coordinates in a first-in, first-out manner.

Images