Efficient fusion RGBT target tracking method based on Mmba architecture

Abstract

The invention discloses an efficient fusion RGBT target tracking method based on a Mama architecture, belongs to the technical field of deep learning, and solves the problem of how to balance the performance and fusion efficiency of multi-modal target tracking. A global-local efficient fusion module is introduced to carry out modal interaction and fusion, two modal features carry out direct information interaction through channel exchange of odd-even indexes, and then further interaction is carried out through multi-path Mama; the multi-path Mamba designs four paths for performing relation modeling on multi-modal data, a forward path and a reverse path for performing modal-level interaction from a global view angle, and a mixed forward and channel forward path for performing token-level and channel-level interaction from a local view angle; a plurality of GLE modules are inserted into the feature extraction and relation modeling trunk to perform template fusion and search frame fusion in different stages, and related information is fed back to a trunk network; according to the invention, through the fusion module designed by the Mama architecture, fewer parameters and higher calculation efficiency are achieved, and meanwhile, higher tracking performance is kept.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal target tracking in deep learning, and relates to an efficient fusion RGBT target tracking method based on Mamba architecture. Background Art

[0002] Object tracking is an important research field in computer vision. Its task is to predict the position and scale of the target in the subsequent frames based on the target in the initial frame. It has broad application prospects in video surveillance processing, autonomous driving, human-computer interaction, motion analysis and other fields. However, in extreme scenes such as rain, fog, and night, tracking based only on visible light (RGB) images becomes fragile, which inspires the combination with thermal infrared (TIR) ​​images to achieve more robust tracking using modal complementary information, namely RGBT tracking. In recent years, RGBT tracking has attracted more and more attention from scholars. Since this task involves two modalities, how to effectively fuse them to utilize the useful information in them has become a challenge. In order to solve the above problems, many works have designed different fusion methods to improve the reliability of tracking. According to the architecture used, the current mainstream RGBT modality fusion methods can be roughly divided into convolutional neural network (CNN)-based fusion, Transformer-based fusion and other fusion.

[0003] The CNN-based fusion method can effectively extract the spatial hierarchical structure features of the image through convolution operations, which is crucial for understanding the image content. Li et al., Challenge-Aware RGBT Tracking. European Conference on Computer Vision. 2020: 222-237. Based on different challenges, modality-sharing challenges and modality-specific challenge branches are designed, and multiple branches are aggregated through convolution. Lu et al., Duality-Gated Mutual Condition Network for RGBT Tracking. IEEE Transactions on Neural Networks and Learning Systems. 2022: 1-14. Convolution is used to bidirectionally condition the discriminative information of one modality to adaptively guide the feature learning of the target appearance in another modality, thereby effectively enhancing the target representation of all modalities.

[0004] The Transformer-based fusion method calculates the correlation between different elements through the self-attention mechanism, can process all elements in the sequence in parallel, and can capture long-distance dependencies. It has been widely used in many tasks in recent years. For example, Xiao et al., Attribute-Based Progressive Fusion Network for RGBTTracking. Proceedings of the AAAI Conference on Artificial Intelligence. 2022: 2831-2838. By challenging the attribute decoupling fusion process, an attribute-based progressive fusion network is proposed, in which Transformer is introduced to construct an enhanced fusion module. Hui et al., Bridging Search Region Interaction with Template for RGB-T Tracking. Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023: 13630-13639. The template is used as a medium to bridge the RGB and TIR search regions to achieve full multimodal fusion.

[0005] Other fusion methods include simple addition, channel concatenation, MLP, etc., which often have fewer parameters. Zhang et al., Learning multi-domain convolutional network for RGB-T visual tracking. "201811th International Congress on Image and Signal Processing, BioMedicalEngineering and Informatics (CISP-BMEI)". 2018: 8633180. RGB and TIR features are fused in a channel cascade manner. Cao et al., Bi-directional Adapter for Multi-modal Tracking. "Proceedings of the AAAI Conference on Artificial Intelligence". 2024: 927-935. A multimodal visual cue tracking model based on a general bidirectional adapter is proposed, which uses MLP and realizes mutual promotion between multimodal through cross-cues.

[0006] The existing fusion methods are mainly designed based on CNN or Transformer architecture. Among them, the CNN architecture has a small receptive field, which makes it difficult to perform global information interaction and fully model the association of multimodal information. The method based on the Transformer architecture needs to calculate the correlation between each pair of tags due to its core self-attention mechanism, which causes its computational overhead to grow quadratically, greatly increasing the training cost. Other simple fusion methods are difficult to have good modeling capabilities. Summary of the invention

[0007] The technical solution of the present invention is used to solve the problem of how to balance the performance and fusion efficiency of multi-modal target tracking.

[0008] The present invention solves the above technical problems through the following technical solutions:

[0009] An efficient fusion RGBT target tracking method based on Mamba architecture adopts ViT Blocks structure and global-local efficient fusion module, wherein the global-local efficient fusion module includes channel exchange and multi-path Mamba architecture; by placing the global-local efficient fusion module after the ViT Block structure, the two-modal features exchange the channels at the odd indexes for information interaction, and use the multi-path Mamba architecture to replace the ViT Self-attention in the Block structure, and retains the original features through residual connections; the multi-path Mamba architecture models the modal relationship from four paths, and the four paths are: the forward path and the reverse path modeling methods scan and model the token from the forward and reverse directions respectively, which are responsible for modeling the information within the modality and performing global, modal-level information interaction on the multi-modal information; the mixed forward path and the channel forward path modeling methods shorten the token distance of the two modalities from the spatial and channel dimensions respectively, and can perform token-level and channel-level modeling between the modalities from a local perspective; the modeling results of the four paths are obtained by linearly projecting the input to obtain the gating parameters, and the gating parameters are added to aggregate different interactive information, and finally the output token sequence is obtained, so as to fully interact with the multi-modal data.

[0010] Furthermore, the bimodal features exchange the channels at odd indices for information interaction as follows: Given the intermediate layer features of RGB and TIR and Where B, N, and D represent the batch size, the number of tokens, and the number of channels, respectively; when the channel is swapped, F x The channels at odd index positions will be f y The channels at the corresponding positions in f are replaced, otherwise, yThe channels at odd index positions will be f x Replace the channels at the corresponding positions in .

[0011] Furthermore, the method of using the multi-path Mamba architecture to replace the Self-attention in the ViT Block structure and retaining the original features through the residual connection is as follows:

[0012] After channel exchange, we get the multimodal feature f u =Concat(f x ,f y ), where Concat represents tensor concatenation in the token dimension;

[0013] Multimodal feature f u Four paths are modeled through Multipath Mamba to achieve more comprehensive multimodal information interaction from more dimensions;

[0014] The design process of using the multi-path Mamba architecture to replace the Self-attention in the ViT Block structure is as follows:

[0015] f′ u =LN(f u )+MPM(LN(f u ))

[0016] f″ u =LN(f′ u )+MLP(f′ u )

[0017] The output f″ u It will be divided into f″ evenly according to the channel dimension u1 and f″ u2 Finally, it will enter the backbone network through the residual connection to introduce modal fusion information, and use f′ x and f′ y Respectively represent the input of RGB and TIR in the next encoder layer, then:

[0018]

[0019] Among them, LN(f u ) indicates the u The function for layer normalization, MPM(LN(f u )) represents the function that is fed into Multipath Mamba for modeling after layer normalization, f′ u represents the intermediate variable, f″ u Represents the output after the global-local efficient fusion module.

[0020] Furthermore, the two path modeling methods of the forward path and the reverse path respectively scan and model the token from the forward and reverse directions as follows:

[0021] The forward path scans and calculates the input in sequence, that is, I f =I, where I represents the input feature, I f Represents the input of the forward path; in the token dimension, the input feature is expressed as:

[0022] I=[I rgb (1), ..., I rgb (n), I tir (1), ..., I tir (n)]

[0023] Among them, I rgb (1) I tir (1) represents the first token of the input RGB and TIR modalities, I rgb (n), I tir (n) represents the nth token of the input RGB and TIR modalities respectively;

[0024] The reverse path performs reverse scanning calculations on the original input to reversely aggregate information, which is more suitable for visual tasks with position information. Its sequence is expressed as:

[0025] I b =[I tir (n), ..., I tir (1), I rgb (n), ..., I rgb (1)]

[0026] Afterwards, 1-D convolution and SSM are used to model the intrinsic modal relationship respectively. Since the input token contains RGB and TIR, it also participates in coarse-grained cross-modal interaction.

[0027] Furthermore, the two path modeling methods of the hybrid forward path and the channel forward path are as follows to shorten the token distance of the two modes from the spatial and channel dimensions respectively:

[0028] In the hybrid forward path computation, the tokens of the two modalities are sorted alternately, and the spatially aligned tokens are clustered together in the token dimension;

[0029] For channel forward path computation, spatially aligned tokens are connected via the channel dimension to jointly model the two modalities in parallel.

[0030] Perform fine-grained cross-modal modeling at the token level by using two scanning mechanisms: hybrid forward path and channel forward path.

[0031] The sequence of the hybrid forward path scanning modeling is expressed as follows:

[0032] I mix =[I rgb (1), I tir (1),…,I rgb (n), I tir (n)]

[0033] The sequence of the channel forward path scanning modeling is expressed as follows:

[0034] I channel =[Concat(I rgb (1), I tir (1)),…,Concat(I rgb (n), I tir (n))]

[0035] Among them, Concat represents the concatenation of tokens in the channel dimension. rgb (1), I tir (1)) indicates that the input RGB and the first token of the TIR modality are concatenated according to the channel dimension. Concat(I rgb (n), I tir (n)) indicates that the input RGB and the nth token of the TIR modality are concatenated according to the channel dimension.

[0036] An electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above-mentioned efficient fusion RGBT target tracking method based on the Mamba architecture, and the processor is configured to execute the program stored in the memory.

[0037] A storage medium stores a computer program, which, when executed by a processor, executes the steps of the above-mentioned efficient fusion RGBT target tracking method based on the Mamba architecture.

[0038] The advantages of the present invention are:

[0039] The present invention is based on the efficient fusion RGBT target tracking method of the Mamba architecture, which is composed of ViT Blocks and a global-local efficient fusion module (GLE), wherein the GLE is mainly composed of two parts: channel swapping (Channel Swapping) and multi-path Mamba (MPM). In order to interact and fuse multimodal information, the present invention proposes a global-local efficient fusion module and places it after a specific ViT Block. The two-modal features will first exchange the channels at the odd indexes for preliminary direct information interaction, and then refer to the structural design of ViT, use multi-path Mamba to replace the original Self-attention, and retain the original features through residual connections; in order to more fully interact with multimodal information, the multi-path Mamba of the present invention models the modal relationship from four paths. It models the modal-level relationship from a global perspective through forward and reverse paths. However, the core SSM of Mamba is modeled in a sequential and causal manner. When multimodal features are introduced as input, the distance between the modalities will be too large regardless of the forward or reverse calculation path, which makes it difficult to perform effective and fine-grained interaction between the modalities. Therefore, the present invention proposes two fine-grained scanning modeling methods for multimodal images, namely, hybrid forward and channel forward path modeling methods, which respectively shorten the token distance between the two modalities from the spatial and channel dimensions to perform token-level and channel-level relationship modeling from a local perspective. The present invention utilizes the new Mamba architecture to reduce the computational overhead of multimodal fusion while maintaining tracking performance, and utilizes multi-path modeling to perform sufficient multimodal interaction to achieve efficient RGBT target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is an overall network framework diagram of the efficient fusion RGBT target tracking method based on the Mamba architecture according to an embodiment of the present invention;

[0041] Figure 2 This is a structural diagram of the global-local efficient fusion module of the efficient fusion RGBT target tracking method based on the Mamba architecture in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0044] Embodiment 1

[0045] like Figure 1 As shown, the efficient fusion RGBT target tracking method based on the Mamba architecture of the embodiment of the present invention includes the following contents:

[0046] 1. Global-local efficient fusion module

[0047] The global-local efficient fusion module is used to exchange the channels at the odd index positions of the two modalities for direct, parameter-free information interaction. In addition, a residual block is formed through the normalization layer, multipath Mamba and MLP for multi-dimensional, full multimodal interaction, and then the fusion information is introduced into the backbone network in the form of addition. The global-local efficient fusion module (GLE) is inserted after a specific layer of the backbone to fully interact with the multimodal tokens of the template and the search area, while introducing as few parameters as possible. Its main components are channel exchange, layer normalization (LN), multipath Mamba and MLP layers.

[0048] The purpose of channel exchange is to conduct preliminary and direct information exchange between multiple modalities without introducing any parameters, thus keeping the model lightweight. Due to the symmetry of the exchange, the mutual exchange at odd indexes is the same as the mutual exchange at even indexes, so here, the channels at odd indexes of the two modalities are exchanged. Specifically, given the intermediate layer features of RGB and TIR and Where B, N, and D represent the batch size, the number of tokens, and the number of channels, respectively. Then when the channel is swapped, f x The channels at odd index positions will be f y The channels at the corresponding positions in f are replaced, otherwise, y The channels at odd index positions will be f x After channel exchange, we get the multimodal feature f u =Concat(f x ,f y ), where Concat represents tensor concatenation in the token dimension. Then the multimodal feature f u Multipath Mamba is used to perform four-path directional modeling to enable more comprehensive multimodal information interaction from more dimensions.

[0049] The process of replacing the Self-attention design in the ViT Block structure with the multi-path Mamba architecture is as follows:

[0050] f′ u =LN(f u )+MPM(LN(f u ))

[0051] f″ u =LN(f′ u )+MLP(f′ u )

[0052] The output f″ u It will be divided into f″ evenly according to the channel dimension u1 and f″ u2 Finally, it will enter the backbone network through the residual connection to introduce modal fusion information, and use f′ x and f′ y Respectively represent the input of RGB and TIR in the next encoder layer, then:

[0053]

[0054] Among them, LN(f u ) indicates the u The function for layer normalization, MPM(LN(f u )) represents the function that is fed into Multipath Mamba for modeling after layer normalization, f′ u represents the intermediate variable, f″ u Represents the output after the global-local efficient fusion module.

[0055] 2. Multipath Mamba (MPM)

[0056] The multi-path Mamba modeled by four-directional scanning is adapted to multimodal tasks and enhances the multimodal interaction capabilities of the model. The forward and backward calculations model the information within the modality and perform global, modal-level information interaction on multimodal information by forward and backward scanning calculations on the input token. At the same time, in order to adapt to and associate multimodal data, two paths, Mix Forward and Channel Forward, are used to bring the spatially aligned tokens closer in the spatial dimension and channel dimension, respectively, so as to perform token-level and channel-level modeling between modalities from a local perspective and more clearly explore the dependencies between modalities.

[0057] A novel multi-path Mamba with four-directional scanning modeling is designed to adapt to multimodal tasks and enhance the multimodal interaction capabilities of the model. In the token dimension, the input features can be represented as:

[0058] I=[I rgb (1),…,Irgb (n), I tir (1),…,I tir (n)]

[0059] Specifically, the forward and backward branches are responsible for modeling the intra-modal information and performing global, modal-level information interaction on multi-modal information. In the forward branch, the input is scanned and calculated sequentially, i.e., I f =I, where I represents the input feature, I f Represents the input of the forward path. In the reverse branch, the original input is scanned and calculated in reverse to aggregate information in reverse, which is more suitable for visual tasks with position information. Its sequence is represented as:

[0060] I b =[I tir (n),…,I tir (1), I rgb (n),…,I rgb (1)]

[0061] Afterwards, 1-D convolution and SSM (State Space Model, which is a mathematical model used to describe and analyze the behavior of dynamic systems. The core idea is to express the relationship between the current state of the system and the input and output with a set of equations) are used to model the intrinsic modal relationship. In addition, since the input token contains RGB and TIR (Thermal Infrared, which is an image formed by a thermal infrared sensor receiving and recording the thermal radiation energy emitted by the target object), it also participates in coarse-grained cross-modal interaction.

[0062] However, the modeling of SSM is sequential and causal. When multimodal features are introduced as input, the distance between modalities will be too large, whether in forward or backward calculation, which makes it difficult for modalities to interact effectively and fine-grained. Therefore, two fine-grained scanning modeling methods for multimodal images are proposed, called hybrid forward and channel forward. In the hybrid forward calculation, the tokens of the two modalities are alternately sorted, and the spatially aligned tokens are clustered together in the token dimension, and these tokens are close in space. For the channel forward calculation, the spatially aligned tokens are connected through the channel dimension to jointly model the two modalities in parallel. These two scanning mechanisms can be used to perform token-level and channel-level modeling between modalities from a local perspective, thereby more explicitly exploring the dependencies between modalities. The sequence representation of these two scanning modeling is as follows:

[0063] I mix =[I rgb (1), I tir (1),…,I rgb(n), I tir (n)]

[0064] I channel =[Concat(I rgb (1), I tir (1)),…,Concat(I rgb (n), I tir (n))]

[0065] Among them, Concat represents the concatenation of tokens in the channel dimension. rgb (1), I tir (1)) indicates that the input RGB and the first token of the TIR modality are concatenated according to the channel dimension. Concat(I rgb (n), I tir (n)) indicates that the input RGB and the nth token of the TIR modality are concatenated according to the channel dimension.

[0066] Finally, the modeling results of the four paths are gated and added together to aggregate different interactive information, and finally the output token sequence is obtained. z is the gating parameter obtained by linear projection of the input.

[0067] As shown in Table 1 below, the present invention compares the performance with the most advanced RGBT tracking method on the GTOT, RGBT210, RGBT234 and LasHeR datasets to evaluate the method of the present invention and verify the effectiveness of the method of the present invention. The present invention selects the most commonly used evaluation indicators of the dataset for quantitative performance evaluation in each dataset. The LasHeR dataset uses PR, NPR, and SR as evaluation indicators, and the other datasets use PR and SR as evaluation indicators. Among them, PR refers to the percentage of frames in which the distance between the center point of the tracking result and the center of the true bounding box is lower than the specified threshold. SR measures the ratio of frames in which the overlapping area of ​​the tracking result box and the true bounding box is higher than the threshold, and then calculates the SR metric by changing the threshold and calculating the area under the result curve (AUC). In order to eliminate the influence of image resolution and target size on the PR metric, PR is normalized to obtain the normalized accuracy rate NPR when evaluating the tracking performance on the LasHeR dataset. It can be seen from the results that the method of the present invention can be applied to the RGBT target tracking dataset, and the results are significantly improved compared with other methods.

[0068] Table 1 Results comparison chart

[0069]

[0070]

[0071] The method of the present invention introduces a global-local efficient fusion module after a specific Transformer block to perform modal interaction and fusion, wherein the two modal features first directly interact with each other through the channel exchange of the odd-even index, and then further interact through multipath Mamba, MLP, layer normalization and residual connection. Among them, multipath Mamba designs four paths to model the relationship between multimodal data, namely, the forward path and reverse path for global modal-level interaction and the mixed forward path and channel forward path for local token-level interaction and channel-level interaction, and finally inserts multiple global-local efficient fusion modules in the feature extraction and relationship modeling trunk to perform template fusion and search frame fusion at different stages, and feeds back relevant information to the backbone network. The fusion module designed by the Mamba architecture of the present invention has fewer parameters and higher computational efficiency, while maintaining a high tracking performance.

[0072] Embodiment 2

[0073] An electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the efficient fusion RGBT target tracking method based on the Mamba architecture in Example 1, and the processor is configured to execute the program stored in the memory.

[0074] Embodiment 3

[0075] A storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the efficient fusion RGBT target tracking method based on the Mamba architecture in embodiment 1 are executed.

[0076] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient fusion RGBT target tracking method based on Mamba architecture, characterized in that: The ViT Blocks structure and the global-local efficient fusion module are adopted, and the global-local efficient fusion module includes channel exchange and multi-path Mamba architecture. By placing the global-local efficient fusion module after the ViT Block structure, the two modal features exchange the channels at the odd indexes for information interaction, and the multi-path Mamba architecture is used to replace the Self-attention in the ViT Block structure, and the original features are retained through residual connection. The multi-path Mamba architecture models the modal relationship from four paths, and the four paths are: the forward path and the reverse path modeling methods scan and model the token from the forward and reverse directions respectively, which are responsible for modeling the information within the modality and performing global and modal-level information interaction on the multi-modal information; the mixed forward path and the channel forward path modeling methods shorten the token distance of the two modalities from the spatial and channel dimensions respectively, and can perform token-level and channel-level modeling between the modalities from a local perspective; the modeling results of the four paths are obtained by linearly projecting the input to obtain the gating parameters, and the gating parameters are added to aggregate different interactive information, and finally the output token sequence is obtained, so as to fully interact with the multi-modal data.

2. The efficient fusion RGBT target tracking method based on Mamba architecture according to claim 1 is characterized in that: The method of exchanging the channels at odd indices for information interaction using the two-modal features is as follows: Given the intermediate layer features of RGB and TIR and Where B, N, and D represent the batch size, the number of tokens, and the number of channels, respectively; then when the channel is exchanged, f x The channels at odd index positions will be f y The channels at the corresponding positions in f are replaced, otherwise, y The channels at odd index positions will be f x Replace the channels at the corresponding positions in .

3. The efficient fusion RGBT target tracking method based on Mamba architecture according to claim 2 is characterized in that: The method of using the multi-path Mamba architecture to replace the Self-attention in the ViT Block structure and retaining the original features through the residual connection is as follows: After channel exchange, we get the multimodal feature f u =Concat(f x ,f y ), where Concat represents tensor concatenation in the token dimension; Multimodal feature f u Four paths are modeled through Multipath Mamba to achieve more comprehensive multimodal information interaction from more dimensions; The design process of using the multi-path Mamba architecture to replace the Self-attention in the ViT Block structure is as follows: f u ′=LN(f u )+MPM(LN(f u )) f u ″=LN(f u ′)+MLP(f u ′) ″″″ The output f u It will be divided into f evenly according to the channel dimension u1 With f u2 , and finally through the residual connection ″ Enter the backbone network to introduce modal fusion information, and use f x and f y Respectively represent the input of RGB and TIR in the next encoder layer, then: Among them, LN(f u ) indicates the u The function for layer normalization, MPM(LN(f u )) indicates layer normalization and then sending ″′ The function for modeling in multipath Mamba, f u represents the intermediate variable, f u Represents the output after the global-local efficient fusion module.

4. The efficient fusion RGBT target tracking method based on Mamba architecture according to claim 3 is characterized in that: The two path modeling methods of the forward path and the reverse path are as follows: The forward path scans and calculates the input in sequence, that is, I f =I, where I represents the input feature, I f Represents the input of the forward path; in the token dimension, the input feature is expressed as: I=[I rgb (1),…,I rgb (n),I tir (1),…,I tir (n)] Among them, I rgb (1) I tir (1) represents the first token of the input RGB and TIR modalities, I rgb (n), I tir (n) represents the nth token of the input RGB and TIR modalities respectively; The reverse path performs reverse scanning calculations on the original input to reversely aggregate information, which is more suitable for visual tasks with position information. Its sequence is expressed as: I b =[I tir (n),…,I tir (1),I rgb (n),…,I rgb (1)] Afterwards, 1-D convolution and SSM are used to model the intrinsic modal relationship respectively. Since the input token contains RGB and TIR, it also participates in coarse-grained cross-modal interaction.

5. The efficient fusion RGBT target tracking method based on Mamba architecture according to claim 4 is characterized in that: The two path modeling methods of the hybrid forward path and the channel forward path are as follows: In the hybrid forward path computation, the tokens of the two modalities are sorted alternately, and the spatially aligned tokens are clustered together in the token dimension; For channel forward path computation, spatially aligned tokens are connected via the channel dimension to jointly model the two modalities in parallel. Perform fine-grained cross-modal modeling at the token level by using two scanning mechanisms: hybrid forward path and channel forward path. The sequence of the hybrid forward path scanning modeling is expressed as follows: I mix =[I rgb (1),I tir (1),…,I rgb (n),I tir (n)] The sequence of the channel forward path scanning modeling is expressed as follows: I channel =[Concat(I rgb (1),I tir (1)),…,Concat(I rgb (n),I tir (n))] Among them, Concat represents the concatenation of tokens in the channel dimension. rgb (1), I tir (1)) indicates that the input RGB and the first token of the TIR modality are concatenated according to the channel dimension. Concat(I rgb (n), I tir (n)) indicates that the input RGB and the nth token of the TIR modality are concatenated according to the channel dimension.

6. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the efficient fusion RGBT target tracking method based on the Mamba architecture as described in any one of claims 1 to 5, and the processor is configured to execute the program stored in the memory.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the efficient fusion RGBT target tracking method based on the Mamba architecture according to any one of claims 1 to 5 are executed.

Images