Multi-modal image tracking method based on multi-frequency attention and space-frequency state space fusion

Abstract

The present application relates to the field of computer vision image tracking, in particular, the present application proposes a kind of multi-modal image tracking method based on multi-frequency attention and space-frequency state space fusion.The method uses upper branch improved space ViT to focus high-frequency information by multi-frequency attention mechanism, lower branch frequency VIT is extracted to multi-modal image first, then low-frequency frequency domain representation is extracted by convolution filtering, generates low-frequency attention map and is dynamically weighted frequency component by hybrid expert module, generates adaptive frequency feature;The feature is injected into the upper branch in the space-frequency Mamba fusion stage, high-frequency-low-frequency collaborative modeling is realized;Finally, the response map is output by the light weight decoder to generate the target position.The present application improves the tracking accuracy and real-time performance in complex environment by high-frequency-low-frequency hierarchical modeling and hybrid expert network dynamic weighting, especially in the key scene of target edge (high frequency) and background structure (low frequency).

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion. Background Technology

[0002] Target tracking, a core capability of intelligent perception systems, aims to continuously and accurately locate and estimate the state of a specified target in a video sequence. In the traditional single-modal RGB image tracking framework, system performance is highly dependent on ambient lighting conditions and visual saliency. However, in real-world complex scenes, targets often face challenges such as strong lighting changes, severe occlusion, rapid movement, interference from similar backgrounds, low-light conditions at night, or extreme weather, making single-visual-modal trackers prone to failure. To overcome these limitations, multimodal target tracking technology has emerged. Its core idea is to integrate information from multiple heterogeneous sensors, utilizing the complementarity, redundancy, and robustness between different modalities to build a more reliable and adaptive perception system. However, although multimodal data provides a richer information source than traditional single-modal data, its effective utilization still faces significant challenges. Therefore, there is an urgent need for an efficient, adaptive, end-to-end trainable multimodal tracking architecture that can achieve cross-modal semantic alignment, dynamic information selection, and long-term temporal state evolution modeling while ensuring low latency and low power consumption.

[0003] Multimodal target tracking technology enhances information by fusing data from two or more heterogeneous sensors, leveraging the complementarity and redundancy between different modalities. For example, infrared images are insensitive to illumination and can effectively capture target thermal radiation at night or in smoky environments; depth or point cloud data provides three-dimensional spatial structure information, helping to distinguish foreground from background; event cameras have high temporal resolution and low latency, making them suitable for high-speed motion scenes; and language descriptions can serve as high-level semantic priors, guiding the model to focus on targets of specific categories or attributes.

[0004] VisionTransformer (VIT) is widely used in object tracking due to its inherently suitable structure for end-to-end training, powerful global modeling capabilities, and flexible cross-modal attention mechanism. Numerous studies have demonstrated that multimodal object tracking systems based on VIT networks can effectively integrate information from different sources such as RGB, depth, and infrared, significantly improving the robustness and accuracy of object tracking in complex scenes. The general workflow of such methods includes data preprocessing, feature extraction, feature fusion, and state prediction. The data preprocessing includes spatial alignment and temporal synchronization of image sequences of the same target in different modalities, as well as standardization of size and resolution to enable the Transformer model to efficiently learn cross-modal feature representations. Feature extraction involves using multiple parallel ViT encoders to patch and embed the input sequences of each modality, and extracting deep semantic features with a global receptive field through a self-attention mechanism. The feature fusion stage effectively integrates the features extracted from different modalities through methods such as cross-attention or weighted fusion to fully utilize the complementary information between modalities. State prediction refers to using the prediction network to output the target's position, scale, or motion state in the current frame based on the fused multimodal features, thereby achieving continuous tracking of the target.

[0005] Currently, multimodal tracking still faces several challenges. Previous multimodal networks often only fused RGB and another auxiliary modal information, and previous work mainly relied on VIT networks.

[0006] Existing ViT self-attention mechanisms, while capturing long-range spatial context information, exhibit time and memory complexity that increases quadratically with the length of the input sequence. This leads to enormous computational overhead in high-resolution video or multi-frame joint processing, limiting real-time performance. Secondly, the significant differences in data distribution across different modalities mean that simple feature concatenation or attention mechanisms are insufficient to fully exploit complementary information between modalities, potentially resulting in poor fusion performance. Furthermore, in low-data or small-sample tracking scenarios, ViT is prone to overfitting and heavily relies on labeled data. Finally, multimodal fusion relying solely on attention mechanisms may ignore the consistency of local structures or dynamic evolution patterns between modalities, leading to redundant fusion information or noise interference. Summary of the Invention

[0007] To address the aforementioned problems, this invention discloses a multimodal image tracking method and system based on multi-frequency attention and spatial-frequency state-space fusion.

[0008] The technical solution adopted is as follows:

[0009] A multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion includes a multimodal image input module, a dual-branch feature extraction module, a Mamba dynamic fusion module, a hybrid expert weighting module, and a tracking result output module.

[0010] The multimodal image input module preprocesses the input multimodal image and uses a parallel coding network to convert the input image into a sequence form through block embedding and positional coding.

[0011] The dual-branch feature extraction module is used to extract the spatial and frequency features of the multimodal image respectively. The upper branch input sequence extracts the spatial features of the sequence through a multi-frequency attention and feedforward network, while ensuring that the network pays more attention to the high-frequency information of the image for subsequent fusion; the lower branch input sequence first extracts spatial features through the same network as the upper branch, then extracts the amplitude spectrum through a convolution module, and generates the corresponding low-frequency attention map and frequency components through the convolution module.

[0012] The hybrid expert weighting module is used to weight the frequency features of the multimodal image and align them with the spatial features. The attention map generated in the above steps provides weights when the frequency components are input into the hybrid expert network, thereby enhancing important frequency information and suppressing secondary frequency information.

[0013] The spatial frequency Mamba dynamic fusion module is used to fuse the frequency features and spatial features of weighted multimodal images. It establishes long-range dependencies between spatial sequences and frequency domain sequences through a selective state space mechanism and outputs fused features.

[0014] The tracking result output module is used to output the prediction result;

[0015] In the above technical solution, the specific steps for extracting and fusing high-level features from the preprocessed multimodal image in the dual-branch feature extraction module, the spatial frequency Mamba dynamic fusion module, and the hybrid expert weighting module are as follows:

[0016] Each modality image sequence is input into an upper-branch backbone network model formed by stacking multiple improved spatial VIT modules. Each spatial VIT module contains a normalized layer (LN), a multi-frequency attention layer (MFA), a feedforward network layer (MLP), and an activation function (GeLu). The feature extraction process of each network module can be represented as follows:

[0017]

[0018] in This indicates an improvement to the output of the VIT module. The shape of the feature is represented by the number of patches and the dimension, respectively, and the index is... Indicates the number of modalities in the input image, i Indicates the backbone network number An improved VIT module. It employs a multi-head self-attention mechanism and an MLP feedforward network. Each part uses residual connections and layer normalization. This input-output superposition prevents the loss or degradation of key target features after multiple transformations in deep networks, ensuring that target information is stably transmitted to the final layer.

[0019] The multi-frequency attention mechanism is the core module of spatial ViT. It enhances attention to key regions of the target through frequency-domain modulation factors, which is crucial for its ability to fuse with frequency information. The standard self-attention representation is:

[0020]

[0021] Where Q, K, and V are the query, key, and value matrices, respectively, all of which are obtained by projecting the input features through different linear methods. This represents the dimension of the query vector. Multi-frequency attention introduces a frequency domain modulation factor M:

[0022]

[0023] in The frequency domain embedding obtained from the lower branch, and the multi-frequency attention can be represented as:

[0024]

[0025] Each modality image sequence is input into a lower-branch backbone network model formed by stacking multiple improved frequency VIT modules. Each frequency VIT module contains a normalized layer (LN), a multi-frequency attention layer (MFA), a feedforward network layer (MLP), an activation function (GeLu), a frequency domain convolution module, and a hybrid expert weighting module. The feature extraction process of each network module can be represented as follows:

[0026]

[0027] in This represents the input multimodal image. It is a 3×3 convolutional module. For Gaussian low-pass filters:

[0028]

[0029] Where u and v are relative coordinates, defining the position of each point on the Gaussian kernel relative to the center point. This represents the standard deviation of the filter.

[0030] A further improvement to this invention uses frequency attention maps and hybrid expert networks to weight frequency components, providing physical-level structural constraints. The specific operation steps are as follows:

[0031] The frequency features of each modality are concatenated and then input into a convolutional network to generate a frequency domain attention map. :

[0032]

[0033] in , Extracting the real and imaginary parts respectively. For the Sigmoid function, ensure The hybrid expert network is designed with N expert networks, each expert processing the input features:

[0034]

[0035] Frequency domain attention map Weights are generated using MLP. :

[0036]

[0037] Frequency characteristics The output, weighted by an expert network, is:

[0038]

[0039] To integrate with the features of the upper branch space, the Adapter module is used here. Mapped to the upper branch feature dimension C:

[0040]

[0041] Output As input for the Mamba fusion phase.

[0042] A further improvement to this invention involves adding a space-frequency Mamba dynamic fusion module every two improved VIT modules in the upper branch space VIT network. The entire process is implemented by selecting the state space, and the specific operation steps are as follows:

[0043] First, spatial features Frequency characteristics By splicing Then, parameters are dynamically generated through a selective state-space mechanism to model long sequence dependencies:

[0044]

[0045]

[0046] in B is the selectivity parameter, C is the state transition parameter, and D is the output parameter. The state space is updated through recursive computation. The output features are mapped to the state space, and finally, the output features are fused through a gating network:

[0047] ;

[0048] in For element-wise multiplication, To output fused features.

[0049] Final output features The response graph R is generated using a lightweight decoder:

[0050]

[0051] Finally, the target location is output:

[0052] ; ;

[0053] Using cross-entropy loss, IOU loss, and The loss function measures the difference between the model's predictions and the true labels:

[0054] ;

[0055] ;

[0056] ;

[0057] Where N is the batch size. For real labels, Let A be the probability of existence predicted by the model; A and B are the predicted bounding box and the ground truth bounding box, respectively; and C is the smallest bounding rectangle containing both. To predict coordinates, These are the actual coordinates.

[0058] The advantages of this invention are:

[0059] 1. The improved spatial Vision Transformer (ViT) with an upper branch focuses on high-frequency information through a multi-frequency attention mechanism;

[0060] 2. The lower branch frequency VIT first extracts features from the multimodal image and then extracts the low-frequency domain representation through convolutional filtering to generate a low-frequency attention map. The frequency components are dynamically weighted through the Hybrid Expert Module (MoE) to generate adaptive frequency features. These features are then injected into the upper branch through the spatial-frequency Mamba fusion stage to achieve high-frequency-low-frequency collaborative modeling.

[0061] 3. Finally, a response map is generated using a lightweight decoder to output the target location. This invention significantly improves tracking accuracy and real-time performance in complex environments through high-frequency-low-frequency hierarchical modeling and dynamic weighting using a hybrid expert network, with particularly outstanding advantages in critical scenarios involving target edges (high frequency) and background structures (low frequency).

[0062] 4. Frequency Domain Two-Branch: The upper branch does not perform Fourier transform and inverse Fourier transform. Instead, it uses the frequency domain modulation factor of multi-frequency attention to make the network focus on high-frequency information. Only the lower branch, the low-frequency branch, uses frequency domain transformation, which introduces information loss.

[0063] 5. Previous works mainly relied on VIT networks, while this application uses Mamba networks in the feature fusion layer. Firstly, it supports real-time tracking of long videos / high resolutions. Secondly, it has low computational complexity and does not require calculating the attention weights of all element pairs. Thirdly, the state selection space allows the model to dynamically adjust the state update strategy according to the current input content, thus solving the problem that the fixed-length attention window of VIT cannot adapt to the fast movement of the target. Attached Figure Description

[0064] Figure 1 This is a flowchart of the present invention.

[0065] Figure 2 This is a model block diagram of the present invention.

[0066] Figure 3(a) is a comparison of the accuracy of the present invention with other models on the Lasher dataset.

[0067] Figure 3(b) is a comparison of the accuracy of the present invention with other models on the Lasher dataset.

[0068] Figure 4(a) is a comparison of the accuracy of the present invention with other models on the Visevent dataset.

[0069] Figure 4(b) is a comparison of the accuracy of the present invention with other models on the Visevent dataset.

[0070] Figure 5 This is a visualization of the tracking results of this invention on the RGB-D dataset.

[0071] Figure 6 This is a visualization of the tracking results of this invention on the RGB-E dataset.

[0072] Figure 7 This is a visualization of the tracking results of this invention on the RGB-T dataset. Detailed Implementation

[0073] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0074] like Figure 1 As shown, see also Figure 2 A method based on a multimodal image input module, a dual-branch feature extraction module, a spatial frequency Mamba dynamic fusion module, a hybrid expert weighting module, and a tracking result output module;

[0075] The multimodal image input module preprocesses the input multimodal image by using a parallel coding network to convert the input image into a sequence form through block embedding and positional coding.

[0076] The dual-branch feature extraction module is used to extract the spatial and frequency features of multimodal images separately. The upper branch input sequence extracts the spatial features of the sequence through a multi-frequency attention and feedforward network, while ensuring that the network pays more attention to the high-frequency information of the image for subsequent fusion. The lower branch input sequence first extracts spatial features through the same network as the upper branch, and then extracts the amplitude spectrum through a convolution module that replaces the FFT. After passing through the convolution module, the corresponding low-frequency attention map and frequency components are generated.

[0077] The hybrid expert weighting module is used to weight the frequency features of the multimodal image and align them with the spatial features. The attention map generated in the above steps provides weights when the frequency components are input into the hybrid expert network, thereby enhancing important frequency information and suppressing secondary frequency information.

[0078] The spatial frequency Mamba dynamic fusion module is used to fuse the frequency and spatial features of weighted multimodal images. It establishes long-range dependencies between spatial and frequency domain sequences through a selective state space mechanism and outputs fused features.

[0079] The tracking result output module is used to output the prediction results;

[0080] This invention proposes a multimodal image tracking method and system based on multi-frequency attention and spatial-frequency state-space fusion. To fully utilize the spatial and frequency information of the image, the improved spatial ViT branch focuses on high-frequency information while extracting spatial features. The lower branch frequency ViT first extracts features from the multimodal image and then extracts low-frequency domain representations through convolutional filtering, generating a low-frequency attention map. This low-frequency feature is then dynamically weighted by a hybrid expert module to generate adaptive frequency features. These features are injected into the upper branch through a spatial-frequency Mamba fusion stage, achieving high-frequency-low-frequency collaborative modeling and fusion together with the upper branch features. Finally, a lightweight decoder generates a response map to output the target position. This method significantly improves tracking accuracy and real-time performance in complex environments through high-frequency-low-frequency hierarchical modeling and MoE dynamic weighting, especially in critical scenarios involving target edges (high frequency) and background structures (low frequency). Extensive experiments have verified the superiority of this method, demonstrating its ability to improve the accuracy and effectiveness of target tracking.

[0081] Specifically, the methods and steps used in the above system are as follows:

[0082] For the input multimodal template image and multimodal search images Preprocessing is performed, where subscripts are used. This indicates the number of modalities in the input image, B, C, , , , It represents the shape of the image frame, and indicates the number of input image frames and the channels, height, and width of the input image, respectively.

[0083] Input processing involves converting the image frames into a sequence using a parallel coding network through block embedding and positional coding, and then stitching them together. , , , The images are input into a feature extractor to obtain the spatial features of each modality image, denoted as . Here, the feature extractor for each branch uses the same network, or different networks can be used, as long as the shape of the spatial features output by different branches is the same, so that these features can be fused and fed into the next module. In this invention, each branch uses an improved spatial VIT encoder model as the feature extractor (a convolutional network can also be used). The spatial VIT encoder consists of a normalization layer LN, a multi-frequency attention layer MFA, a feedforward network layer MLP, and an activation function GeLu. The multi-head self-attention mechanism and the MLP feedforward network each use residual connections and layer normalization. Therefore, the feature extraction process can be represented as:

[0084]

[0085] in This indicates an improvement to the output of the VIT module, i Indicates the backbone network number An improved VIT module is proposed. This input-output stacking approach prevents the loss or degradation of key target features after multiple transformations in deep networks, ensuring that target information is stably transmitted to the final layer. It also avoids gradient vanishing, allowing the network to successfully learn subtle changes in the target against complex backgrounds, thereby improving tracking robustness. To ensure the progressive extraction and enhancement of image feature representations, enabling the model to learn visual information from local to global and from simple to complex, this invention uses a 12-layer improved VIT module for progressive feature extraction; the number of layers can be adjusted as needed.

[0086] Next branch will , , ..., The input convolutional network is used to extract frequency domain features. Traditionally, this would involve performing Discrete Wavelet Transform (DWT) or Fast Fourier Transform (FFT) on the spatial features. However, considering that some devices cannot perform FFT, this invention uses an alternative convolutional module to obtain the amplitude spectrum or subband coefficient representation, which is then further processed. However, Fourier Transform introduces additional computational overhead, and its computational characteristics are incompatible with the hardware architecture. Therefore, this invention uses standard convolution to directly capture each frequency component in the spatial domain and generate an attention map. Since the upper branch deals with visible light imaging, its information characteristics make it best suited for capturing local details, so the upper branch focuses on high-frequency information. The lower branch processes other modalities corresponding to global structure, background, and overall trends, such as scene layout in depth maps and temperature distribution in thermal imaging, and should focus on low-frequency information.

[0087] The detailed decomposition process of focusing on different frequency information in the dual-branch selection is as follows:

[0088] First, the frequency components mentioned above are used to extract the low-frequency components in the frequency domain using a low-pass filter:

[0089]

[0090]

[0091] The low-pass filter here occupies 1 / 4 of the central frequency range. (Control the filter width) to ensure that the background structure is preserved while avoiding over-smoothing. Then, the low-frequency components are fed into a convolutional network to obtain a low-frequency attention map. :

[0092]

[0093] It is worth noting that since the output of the convolution module that replaces FFT is also a complex number, the real part (amplitude) represents energy and the imaginary part (phase) represents position information, and both are indispensable. This is not a regular attention map, but rather an intensity map of low-frequency structures, used to guide the weight allocation of the hybrid expert network below. The upper branch obtains high-frequency information by subtracting low-frequency information from the aforementioned frequency components, and uses a multilayer perceptron to generate a frequency tuning factor for multi-frequency attention, enabling the network to implicitly focus on high-frequency information in the image. The collective operation is as follows:

[0094]

[0095]

[0096] In the next branch, to further enhance the low-frequency features, frequency features are obtained. low-frequency components Input a hybrid expert network. The hybrid expert network employs multiple MLP networks, each learning different frequency feature patterns; for example, expert 1 focuses on low-frequency infrared, and expert 2 focuses on high-frequency RGB. The network weights are then derived from the low-frequency attention map described above. Generation, weight The processing procedure is as follows:

[0097] , ;

[0098]

[0099] Since the output dimension M of the hybrid expert network may not be consistent with the feature dimension N of the upper branch VIT space, this invention uses an adapter to achieve feature space alignment. Mapped to the upper branch feature dimension C:

[0100]

[0101] To fully integrate the feature information of the upper and lower branches, this invention employs a Mamba dynamic fusion module for adaptive fusion. In the stacked backbone network, the earlier VIT modules focus more on local details, while later modules gradually integrate a wider range of contextual information to form a global representation. Therefore, this invention performs feature fusion every two VIT modules, which can preserve local details while gradually introducing higher-level semantics and global information, thereby improving the tracker's robustness to changes in the target's appearance. First, the features of the upper and lower branches are concatenated to obtain a joint feature sequence u. Then, dynamic parameters of the state space are generated: state decay rate. Input contribution matrix State output matrix Then, the state space is updated through recursive operations:

[0102] ;

[0103] Finally, the historical state and the current input are combined to output the features at the current time step. To address the noise that feature fusion may introduce, a gating mechanism adaptively suppresses noise. The core of feature conflict resolution utilizes a gating mechanism for dynamic feature fusion.

[0104] ;

[0105] Finally, a 1×1 fully connected convolutional layer is used to compress the C-dimensional features into a 1-channel response map R, directly outputting the target confidence score and bounding box value, location:

[0106] ;

[0107] ;

[0108] This invention uses cross-entropy loss, IOU loss, and L1 loss functions to measure the difference between the model's predicted results and the true labels:

[0109] ;

[0110] ;

[0111] ;

[0112] To verify the effectiveness of the proposed method, the algorithm was compared with more than ten state-of-the-art methods on four multimodal datasets, including VIPT, ProTrack, SPT, OSTrack, DET, DDIMP, DMTracker, CMPP, SGT, FANET, and APFNET. All networks were implemented in the PyTorch framework, trained using the Adam optimizer with a learning rate of 0.0001 and a batch size of 16. The performance of the proposed method was verified and tested using the VisEvent, RGBT-234, DepthTrack, and Lasher datasets. The VisEvent dataset, based on data acquired from event cameras, features extremely high temporal resolution and low latency, containing image sequences recorded as event streams, along with corresponding grayscale frames. The RGBT-234 dataset contains video data in both visible light and thermal infrared (TIR) ​​modalities, comprising 234 video sequences and over 200,000 frames of labeled data. The DepthTrack dataset contains RGB images and depth maps for target tracking in 3D space. Compared to traditional RGB tracking, it adds depth information, which helps handle problems such as occlusion, scale changes, and background interference. The LasHeR dataset integrates multimodal information from RGB images, thermal imaging, and LiDAR. It is one of the few heterogeneous modal fusion datasets available, emphasizing the complementary advantages between heterogeneous sensors. The trained tracking model is used to perform tracking tests on test images in the test dataset.

[0113] The tracking accuracy and scores of each tracking method on the Depth Track and RGB-T234 test datasets are shown in Tables 1 and 2. The best performance is indicated in bold.

[0114] Table 1

[0115]

[0116] Table 2

[0117]

[0118] As can be seen from Tables 1 and 2, and Figures 3(a), 3(b), 4(a), and 4(b), the multimodal target tracking method and system proposed in this invention based on multi-frequency attention and spatial frequency Mamba state space achieves higher accuracy than most existing methods, and achieves competitive performance compared to advanced methods. Furthermore, Figure 5 , 6 Figures 7 and 8 present the visual tracking results of the present invention and other tracking methods on three test set images.

[0119] Quantitative and qualitative results show that, compared with other methods, the present invention can be used for multimodal image tracking.

[0120] This invention proposes a multimodal target tracking method and system based on multi-frequency attention and a spatial frequency Mamba state space. It achieves end-to-end training, with frequency domain features directly serving the tracked target, rather than relying on general image processing. A state space model is introduced to achieve efficient cross-modal interaction with linear complexity. This invention significantly improves tracking accuracy and real-time performance in complex environments through high-frequency-low-frequency hierarchical modeling and dynamic weighting using a hybrid expert network.

[0121] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion, characterized in that, Specifically, the steps include the following: Step 1: Receive the input multimodal image through the multimodal image input module; Step 2: Extract spatial and frequency features of the multimodal image using a dual-branch feature extraction module; extract spatial features of the upper branch input sequence through multi-frequency attention and a feedforward network. The input sequence of the lower branch first extracts spatial features through the same network as the upper branch, then extracts the amplitude spectrum through the convolution module, and generates the corresponding low-frequency attention map and frequency components through the convolution module. Step 3: Use a hybrid expert weighting module to weight the frequency features of the multimodal image and align them with the spatial features; Step 4: The Mamba dynamic fusion module fuses the frequency features and spatial features of the multimodal image; Step 5: The tracking result output module is used to output the prediction result.

2. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 1, characterized in that, The specific process for extracting spatial features in step 2 is as follows: Each modality image sequence is input into an upper-branch backbone network model formed by stacking multiple improved spatial VIT modules. Each spatial VIT module contains a normalized layer (LN), a multi-frequency attention layer (MFA), a feedforward network layer (MLP), and an activation function (GeLu). The feature extraction process of each network module can be represented as follows: ； in This indicates an improvement to the output of the VIT module. The shape of the feature is represented by the number of patches and the dimension, respectively, and the index is... Indicates the number of modalities in the input image, i Indicates the backbone network number An improved VIT module.

3. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 2, characterized in that, The specific steps for extracting frequency domain features in step 2 are as follows: Step 1: Input the extracted spatial features into a convolutional network to extract the frequency domain components from the spatial feature map, and retain low-frequency information using a low-frequency filter. Step 3: The upper branch obtains high-frequency information by subtracting low-frequency information from the frequency components in Step 1, and uses a multilayer perceptron to generate a frequency tuning factor for multi-frequency attention, so that the network implicitly focuses on the high-frequency information of the image.

4. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 3, characterized in that, The frequency domain feature extraction process is implemented by a convolution module: First, spatial features are obtained. Then, the input is a convolutional module and a low-pass filter, represented as: ； ； This represents the input multimodal image. It is a 3×3 convolutional module. It is a Gaussian low-pass filter; Then, the low-frequency information of the frequency features is further mined, and the frequency features of each modality are concatenated and input into a convolutional network to generate a frequency domain attention map. : 。 5. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 4, characterized in that, Step 3, in the hybrid expert weighting module, outputs the enhanced low-frequency features by: processing the frequency domain attention map... Weights generated by the feedforward network layer MLP Each expert processes input features Weighted output frequency characteristics : ； ； ； In order to integrate with the features of the upper branch space, After the adapter aligns the spatial feature dimension C, the aligned features are... Represented as: 。 6. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 5, characterized in that, Step 4: In the spatial frequency Mamba dynamic fusion module, spatial features are... Frequency characteristics By splicing Then, parameters are dynamically generated through a selective state-space mechanism to model the long sequence dependency fusion space and frequency characteristics. ； ； ； B is the state decay rate, C is the state transition parameter, and D is the output parameter. The state space is updated through recursive computation. The output features are mapped to the state space.

7. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 5, characterized in that, Introducing gating to achieve adaptive fusion and ; ； For element-wise multiplication, To output fused features.

8. The multimodal image tracking method based on multi-frequency attention and spatial-frequency state-space fusion according to claim 6, characterized in that, Step 5 involves the tracking result output module, which passes through a convolutional module and a fully connected layer to obtain the predicted channel response map R, the target location pos, and the final bounding box value. : ； ； Using cross-entropy loss IOU loss and The loss function measures the difference between the model's predictions and the true labels: ； ； ； in For batch size, For real labels, Let A be the existence probability predicted by the model; A and B be the predicted bounding box and the ground truth bounding box, respectively; and C be the smallest bounding rectangle containing both. The degree of overlap between the predicted bounding box and the ground truth bounding box; To predict coordinates, These are the actual coordinates.

Images