Single-flow RGB-D target tracking method based on state space model

Abstract

The invention discloses a uniflow RGB-D target tracking method based on a state space model, which belongs to the technical field of computer vision, integrates RGB modal image data and Depth modal image data in a Token form, and performs feature extraction and first-stage fusion through a Mama uniflow backbone network to obtain a first-stage image and a second-stage image; and second-stage fusion is carried out through a multi-modal Mix Mama deep fusion module (3M module), the expression ability and tracking efficiency of cross-modal features are improved, and a prediction result is output by using a tracking head. Through the above mode, the problems of large parameter quantity, low fusion efficiency, noise amplification and the like in a traditional RGB-D target tracking method are successfully solved through Token-level fusion, a Mama single-flow backbone network and an innovative 3M module, and the efficiency and robustness of a tracker are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a single-stream RGB-D target tracking method based on a state-space model. Background Art

[0002] The task of single-target visual tracking is to accurately track the position of a target in subsequent frames, given the target information in the first frame. Visual object tracking (VOT) is of great significance in the field of computer vision and is widely used in fields such as autonomous driving and safety monitoring. However, traditional RGB object tracking has limited performance in complex scenes, such as those with occlusion and extreme lighting conditions. In recent years, researchers have attempted to address the shortcomings of RGB by introducing other modalities (such as depth and thermal infrared), promoting the development of RGB-D object tracking.

[0003] However, existing RGB-D object tracking methods still suffer from issues such as large parameter counts, inefficient fusion, and noise amplification. Furthermore, the Transformer-based two-stream tracking framework not only suffers from a large number of training parameters but also suffers from slow inference speed, making it a practical challenge. Therefore, a simpler and more efficient modality fusion solution is needed. The All Mamba framework significantly reduces training parameters and boasts extremely high inference speed, making it an excellent choice for engineering practice.

[0004] Based on this, the present invention designs a single-stream RGB-D target tracking method based on the state-space model to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a simple, compact, and efficient single-stream RGB-D object tracking method based on the All Mamba architecture, called AMTrack. By integrating token-level fusion, a Mamba single-stream backbone network, and an innovative 3M module, it significantly improves the efficiency and robustness of cross-modal feature fusion, making it applicable to a variety of RGB-D object tracking scenarios.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A single-stream RGB-D object tracking method based on a state-space model includes the following steps:

[0008] Step 1: For each frame of the video sequence, simultaneously obtain the RGB modality image and the depth modality image, and divide the image into the search area image and the initial template image respectively; tokenize each image separately, then splice it into a multimodal input token sequence, and send the multimodal input token sequence to the linear layer for feature embedding;

[0009] Step 2: The multimodal input token sequence is fed into the Mamba single-stream backbone network, where it undergoes normalization and then enters the backbone network, which consists of multiple stacked Mamba blocks. Within each Mamba block, the multimodal input token sequence is fed simultaneously as a single stream, and the backbone network performs feature extraction and first-stage modality fusion.

[0010] Step 3: The features output by the backbone network are input to the 3M module, first normalized, and then decomposed into RGB branch features through the linear layer and Depth branch features ;RGB branch features After convolution and activation function, it is sent to the MSSM module as the main input; Depth branch feature It is directly retained and sent to the MSSM module as auxiliary input after convolution and activation function;

[0011] Step 4: Input the features output by the 3M module fusion into the tracking head for target prediction.

[0012] Furthermore, in step 1, each image is token-encoded separately and then concatenated into a multimodal input token sequence. The specific steps include:

[0013] First, each search area image and each initial template image are divided into non-overlapping or overlapping small blocks to obtain a fixed number of image blocks;

[0014] Afterwards, each image block is mapped into a fixed-length token through linear projection. Each search area image and each initial template image consists of several tokens.

[0015] Next, for the same modality, the tokens corresponding to the search area image and the initial template image are stacked sequentially to obtain a complete modality token sequence;

[0016] Finally, the RGB modal Token sequence and the Depth modal Token sequence are concatenated in the Token dimension to form a fused multimodal input Token sequence.

[0017] Furthermore, in step 1, the multimodal input token sequence is fed into a linear layer for feature embedding. The specific steps include: feeding the multimodal input token sequence into one or more linear layers, and through feature dimensionality upscaling or downscaling, making the token features reach the number of input channels required by the Mamba single-stream backbone network.

[0018] Furthermore, the MSSM module performs the following calculations: ; ; in, is the RGB branch feature that has been preprocessed for the first time, is the Depth branch feature that has been preprocessed for the first time, SiLU is the activation function, Conv is the convolution layer, Linear is the linear layer, and Norm is the normalization operation; ; ; in, is the RGB branch feature that has been preprocessed for the second time; is the Depth branch feature that has been preprocessed for the second time; ; ; in, is the final mixed output of the RGB branch, is the final mixed output of the Depth branch, It is a multimodal state space module.

[0019] Furthermore, the state space unit establishes long-range dependencies through recursive state updates, specifically the update and output of the hidden state h, as follows:

[0020]

[0021]

[0022] in, Hidden state parameters for this layer, Hide for the previous layer, is the input image feature; To output image features, A, B, and C are learnable parameters, and D is the connectivity matrix that provides features connecting input and output.

[0023] Furthermore, the training process uses a joint loss function to optimize the detection and positioning accuracy of the final target. The formula is as follows:

[0024]

[0025] in, is the cross entropy loss, is the L1 loss, is the GIoU loss, 、 、 The values ​​are 5, 1 and 2 respectively.

[0026] In order to better achieve the purpose of the present invention, the present invention also provides a single-stream RGB-D target tracking system based on a state-space model, comprising:

[0027] Token generation and feature embedding module: Encodes RGB modality images and depth modality images into tokens respectively, concatenates them into a multimodal input token sequence, and embeds them into the feature space through a linear layer;

[0028] Mamba single-stream backbone network: Multimodal input token sequences are input in a single stream. The Mamba single-stream backbone network uses a multi-layer Mamba structure for feature extraction and first-stage fusion.

[0029] 3M module: performs the second stage fusion of the deep features output by the Mamba single-stream backbone network and decomposes them into RGB branch features through a linear layer and Depth branch features ,The MSSM module uses the RGB branch as the main input and the Depth branch as the auxiliary input to perform information interaction between the main and auxiliary modal features;

[0030] Tracking head: performs multi-task prediction on the fused features and outputs classification heatmaps and bounding box prediction results;

[0031] Training optimization module: uses a joint loss function to optimize the entire network to improve the accuracy of target detection and positioning.

[0032] In order to better achieve the purpose of the present invention, the present invention also provides a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the single-stream RGB-D target tracking method based on the state-space model.

[0033] Compared with the existing technology, the present invention has the following advantages: 1. The Mamba single-stream backbone network adopts an efficient sequence modeling structure with linear complexity, which can efficiently process large-scale token sequences; the long-range dependency modeling capability is suitable for visual tasks with strong spatial-temporal correlation, such as target tracking; the full-process single-stream input realizes the natural fusion of modal features from the bottom layer, avoiding information loss caused by premature or late fusion; and greatly improves tracking efficiency and model compactness.

[0034] 2. The core MSSM module of the 3M module accepts deep features from both the primary modality (e.g., RGB) and the secondary modality (e.g., depth) as input, flexibly regulating the contribution of features from different modalities to the hidden state and output, enabling efficient, non-saturated cross-modal fusion. With its low parameter count and training-friendly nature, the 3M module improves cross-modal feature representation and fusion capabilities, significantly reduces noise interference, and enhances model robustness.

[0035] 3. Through token-level fusion, Mamba single-stream backbone network and innovative 3M module, this invention successfully solves the problems of large number of parameters, inefficient fusion and noise amplification in traditional RGB-D target tracking methods, and significantly improves the efficiency and robustness of the tracker.

[0036] 4. Experiments have shown that the method of the present invention performs well in multiple benchmark tests, outperforming existing technologies in terms of performance, speed and generalization ability, meeting the requirements of real-time tracking, and has good versatility and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0038] Figure 1 This is a framework diagram of the single-stream RGB-D target tracking method of the present invention.

[0039] Figure 2 Schematic diagram of the paradigm differences between our present invention and common object tracking methods. (a) shows the existing dual-stream tracking framework, (b) shows the existing hint learning tracking framework, (c) shows the existing single-stream ViT tracking framework, and (d) shows the single-stream All Mamba tracking framework of our present invention.

[0040] Figure 3 This is the flowchart of the 3M module.

[0041] Figure 4 : These are visualization graphs of the target classification scores after the two-stage fusion of the present invention. The “#” in the graph represents the frame number of the tested video.

[0042] Figure 5 Visualization of the comparison between the tracking results of the present invention and the existing algorithm Figure 1 (Adapter_Indoor Scene).

[0043] Figure 6 Visualization of the comparison between the tracking results of the present invention and the existing algorithm Figure 2 (Backpack_Interior Scene).

[0044] Figure 7 Visualization of the comparison between the tracking results of the present invention and the existing algorithm Figure 3 (Duck_Outdoor Scene).

[0045] Figure 8 A bubble chart comparing the performance-efficiency of the present invention and existing algorithms. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] Example 1: Please refer to the accompanying drawings of the specification Figure 1 , a single-stream RGB-D object tracking method based on the state-space model, named AMTrack, includes the following steps:

[0048] Step 1: Token generation and feature embedding

[0049] 1.1 Multimodal Image Acquisition and Preprocessing

[0050] For each frame of the video sequence, both the RGB modality image and the depth modality image are obtained. For the tracking task, the input image is divided into two types of regions:

[0051] Search Region (SR): The current frame is cropped after expanding a certain proportion based on the target position in the previous frame. It includes the target and its surroundings to ensure that the target is not lost due to drift.

[0052] Initial Template Image (IT): It is the precise cropping result of the target in the first frame or initialization frame, which is used to provide a target reference for subsequent tracking.

[0053] For each frame, a search region image and an initial template image are extracted from the RGB and depth modal images, resulting in four input image regions. All images are resized to a fixed size (e.g., 128×128) and normalized (using mean-variance normalization, etc.) to meet the subsequent Mamba single-stream backbone network input requirements.

[0054] 1.2 Token encoding and multimodal feature concatenation

[0055] First, each search area image and each initial template image are divided into non-overlapping or overlapping small blocks (such as 16×16 patches) to obtain a fixed number of image blocks.

[0056] Each image block is then mapped into a fixed-length feature vector (Token) through linear projection (e.g., a fully connected layer). Each search region image and each initial template image consists of several Tokens. This encodes the input image into a feature form of a Token sequence for easy subsequent use.

[0057] Next, for the same modality (RGB or Depth), the tokens corresponding to the search area image and the initial template image are stacked sequentially to obtain a complete modality token sequence.

[0058] Finally, the RGB modal Token sequence and the Depth modal Token sequence are concatenated in the Token dimension to form a fused multimodal input Token sequence.

[0059] 1.3 Feature Embedding

[0060] The multimodal input token sequence is fed into one or more linear layers, where, through feature dimensionality upscaling or downscaling, the token features are adjusted to meet the input channel count required by the Mamba single-stream backbone network. Furthermore, position embedding can be added to this process to enhance the ability to express target position information.

[0061] Step 2: Mamba single-stream backbone network performs feature extraction and first-stage fusion

[0062] The multimodal input token sequence is fed into the Mamba single-stream backbone network, where it first undergoes normalization (e.g., LayerNorm) to maintain numerical stability. It then enters the backbone network, consisting of multiple layers (e.g., L layers) of stacked Mamba blocks, to complete feature extraction and the first stage of modal fusion.

[0063] In each Mamba Block, all multimodal input token sequences (including RGB and DepthToken) are input synchronously in a single stream. During the sequence modeling process (referring to the image feature extraction process), the first stage of modal fusion and global context modeling (referring to the process of establishing relationships between features) are completed simultaneously. Each Mamba Block (bidirectional Mamba encoder) contains the following main units:

[0064] Conv1d: A feature extraction unit composed of a simple convolutional structure, used in features in different directions, and is divided into forward and backward Conv1d.

[0065] Forward SSM and Backward SSM: Both are composed of the core component, the State Space Unit (SSM). The forward SSM uses the image token as input for forward encoding, while the backward SSM uses reverse encoding for encoding. This allows image features to be arranged in two directions during input.

[0066] State Space Unit (SSM): The core component, which establishes long-range dependencies through recursive state updates. Specifically, it updates and outputs the hidden state h, using learnable parameters A, B, and C. The formula is as follows:

[0067]

[0068]

[0069] in, Hidden state parameters for this layer, Hide for the previous layer, is the input image feature; To output image features, A, B, and C are learnable parameters, and D is the connectivity matrix that provides features connecting input and output.

[0070] Linear projection layer: used to adjust feature dimensions.

[0071] Residual connections and layer normalization (LayerNorm): prevent gradient disappearance and accelerate training convergence.

[0072] Activation function (SiLU) and Dropout (regularization): Improve nonlinear expression capabilities and enhance model generalization.

[0073] The single independent output of each layer of bidirectional Mamba encoder is .

[0074] Unlike traditional convolutional or self-attention architectures, the Mamba single-stream backbone network uses the SSM mechanism to enable explicit information transfer between tokens at very long distances, significantly enhancing the ability to discriminate between objects and backgrounds, and across modalities. After stacking multiple layers, the feature sequence output by the Mamba single-stream backbone network already contains rich spatial, contextual, and preliminary cross-modal information, laying the foundation for deeper multimodal fusion.

[0075] The Mamba single-stream backbone network adopts an efficient sequence modeling structure with linear complexity, which can efficiently process large-scale token sequences. Its long-range dependency modeling capabilities are suitable for visual tasks with strong spatial-temporal correlation, such as target tracking. Its full-process single-stream input achieves natural fusion of modal features from the bottom layer, avoiding information loss caused by premature or late fusion, significantly improving tracking efficiency and model compactness.

[0076] Step 3: Multimodal Mix Mamba deep fusion module (3M module) performs the second stage of fusion

[0077] To further enhance the cross-modal interaction of RGB and Depth features, the present invention designs a low-parameter and efficient multimodal Mix Mamba deep fusion module for the second-stage fusion after the backbone network.

[0078] As attached Figure 3 As shown in Figure 2, the features output by the L-layer backbone network are input to the 3M module, first normalized, and then decomposed into two independent features through the linear layer: RGB branch features and Depth branch features .

[0079] RGB branch features After convolution and activation function, it is sent to the MSSM module (Multimodal State Space Model) as the main input;

[0080] Depth branch characteristics The image is directly retained for RGB self-modal completion and Depth cross-modal complementation. After convolution and activation function, it is sent to the MSSM module as auxiliary input.

[0081] The MSSM module uses the RGB branch as its primary input and the Depth branch as its secondary input. The state transition matrices B and C in the MSSM module are associated with the features of the secondary modality (Depth modality), while the output is associated with the features of the primary modality (RGB modality).

[0082] The MSSM module performs the following calculations:

[0083] ; ;

[0084] in, is the RGB branch feature that has been preprocessed for the first time, is the Depth branch feature that has been preprocessed for the first time, SiLU is the activation function, Conv is the convolution layer, Linear is the linear layer, and Norm is the normalization operation;

[0085]

[0086]

[0087] in, is the RGB branch feature that has been preprocessed for the second time; is the Depth branch feature that has been preprocessed for the second time;

[0088]

[0089]

[0090] in, is the final mixed output of the RGB branch, is the final mixed output of the Depth branch, It is a multimodal state space module;

[0091] The intermediate results after fusion output are multiplied with RGB and Depth modal features respectively and then added to obtain the final fusion feature 、 , to achieve efficient cross-modal information fusion and avoid the noise amplification problem caused by traditional saturated interaction.

[0092] The MSSM module at the core of the 3M module accepts deep features of the main modality (such as RGB) and the auxiliary modality (such as depth) as input, flexibly controls the contribution of different modal features to the hidden state and output results, and realizes efficient and non-saturated cross-modal fusion.

[0093] The 3M module has low parameters and is training-friendly. It improves cross-modal feature expression and fusion capabilities, significantly reduces noise interference, and enhances model robustness.

[0094] Step 4: Tracking head and joint loss function design

[0095] The features fused from the 3M module are fed into the tracking head. The tracking head consists of multiple sub-convolutional networks, each responsible for generating the target's classification score, bounding box coordinates, and offset, and outputting classification heatmaps and bounding box predictions to predict the target's location and category.

[0096] The training process uses a joint loss function to optimize the detection and positioning accuracy of the final target. The formula is as follows:

[0097]

[0098] in, is the cross entropy loss, which is used to optimize the classification accuracy; L1 loss is used to optimize the regression accuracy of the bounding box; is the GIoU (Generalized Intersection-over-Union) loss, which is used to further improve the positioning robustness of the bounding box. 、 、 The values ​​are 5, 1 and 2 respectively.

[0099] Example 2: A single-stream RGB-D target tracking system based on a state-space model, comprising:

[0100] Token generation and feature embedding module: Encode the RGB modality image and the depth modality image into tokens respectively, splice them into a multimodal input token sequence, and embed them into the feature space through a linear layer.

[0101] Mamba single-stream backbone network: Multimodal input token sequences are input in a single stream. The Mamba single-stream backbone network uses a multi-layer Mamba structure for feature extraction and first-stage fusion.

[0102] 3M module: performs the second stage fusion of the deep features output by the Mamba single-stream backbone network. Specifically, it is decomposed into RGB branch features through the linear layer. and Depth branch features ,The MSSM module uses the RGB branch as the main input and the Depth branch as the auxiliary input to ,perform information interaction of the main and auxiliary modal features.

[0103] Tracking head: performs multi-task prediction on the fused features, outputs classification heatmaps and bounding box prediction results, and realizes target location and category prediction.

[0104] Training optimization module: Use the joint loss function (cross entropy, L1, GIoU) ​​to optimize the entire network to improve the accuracy of target detection and positioning. The formula is as follows:

[0105]

[0106] Furthermore, AMTrack can flexibly adapt to scenarios with varying computational resource constraints and target sequence lengths by adjusting the depth or token length of the 3M module. For specific application scenarios, different update intervals or fusion weights can be set to achieve adaptive response to rapidly changing targets.

[0107] Through token-level fusion, Mamba single-stream backbone network and innovative 3M module, this paper successfully solves the problems of large number of parameters, inefficient fusion and noise amplification in traditional RGB-D target tracking methods, and significantly improves the efficiency and robustness of the tracker.

[0108] Embodiment 3: A computer storage medium, wherein the computer storage medium is used to store program data. When the program data is executed by a computer, it is used to implement the single-stream RGB-D target tracking method based on the state-space model described in Embodiment 1.

[0109] Experimental example

[0110] The performance of the proposed method (AMTrack) is compared with other RGB and RGB-D target tracking methods. The results are shown in Tables 1 and 2.

[0111] Pr stands for precision, which indicates the proportion of samples predicted as positive that are actually positive. Re stands for recall, also known as the recall rate, which indicates the proportion of positive samples predicted as positive to the total number of positive samples. F-score is a weighted average of precision and recall; a higher F-score indicates a more robust model. AMTrack-S indicates that the backbone network uses the Vim-S model, while AMTrack-M indicates that the backbone network uses the Vim-M model.

[0112] Table 1 Performance comparison results of the proposed method and other RGB and RGB-D object tracking methods on different test sets

[0113]

[0114] In the table: DAL - A Deep Depth-aware Long-term Tracker (Depth-aware long-term tracker); DeT — DepthTrack: Unveiling the Power of RGBD Tracking SPT — RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking ViPT—Visual Prompt Multi-Modal Tracking; VADT——Visual Adapt for RGBD Tracking.

[0115] As can be seen from Table 1, the proposed method is significantly better than other RGB-D target tracking methods, indicating that the single-stream two-stage fusion model design is more robust for target tracking in complex scenes.

[0116] Table 2 Comparison of training amount, parameter amount and speed of the proposed method with other RGB and RGB-D object tracking methods on different test sets

[0117]

[0118] As can be seen from Table 2, the speed and number of parameters of the proposed method are significantly better than other RGB-D object tracking methods, and robust results can be achieved by training on smaller and fewer datasets.

[0119] Figure 4 It can be seen that after the first stage of fusion, the image feature score map can be better concentrated around the tracked target, and after the second stage of fusion, the tracked target features are more significant.

[0120] Figure 5-Figure 7 It can be seen that the algorithm of the present invention has higher tracking accuracy and is closer to the actual object frame.

[0121] This paper selects the existing algorithms TABBTrack, UBPT, ViPT, DepthRefiner, DeT, and SPT for comparison. The above six algorithms all belong to the field of RGB-D object tracking and have been published in authoritative journals or international conferences, which means that they have commonalities in processing object tracking, making them recognized comparison benchmarks. Figure 8 It can be seen that the red dotted line indicates that the comprehensive performance and speed of the existing algorithms are both lower than that of the present invention. The algorithm of the present invention not only has the highest EAO value but also outperforms the other algorithms in terms of algorithm speed.

[0122] The above experimental results show that the method of the present invention performs well in multiple benchmark tests, outperforms existing technologies in terms of performance, speed and generalization ability, meets the requirements of real-time tracking, and has good versatility and transferability.

[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A single-stream RGB-D target tracking method based on a state-space model, characterized in that: The following steps are involved: Step 1: For each frame of the video sequence, simultaneously obtain the RGB modality image and the depth modality image, and divide the image into the search area image and the initial template image respectively; tokenize each image separately, then splice it into a multimodal input token sequence, and send the multimodal input token sequence to the linear layer for feature embedding; Step 2: The multimodal input token sequence is fed into the Mamba single-stream backbone network, where it undergoes normalization and then enters the backbone network, which consists of multiple stacked Mamba blocks. Within each Mamba block, the multimodal input token sequence is fed simultaneously as a single stream, and the backbone network performs feature extraction and first-stage modality fusion. Step 3: The features output by the backbone network are input to the 3M module, first normalized, and then decomposed into RGB branch features through the linear layer and Depth branch features ;RGB branch features After convolution and activation function, it is sent to the MSSM module as the main input; Depth branch feature It is directly retained and sent to the MSSM module as auxiliary input after convolution and activation function; Step 4: Input the features output by the 3M module fusion into the tracking head for target prediction.

2. The single-stream RGB-D target tracking method based on the state space model according to claim 1 is characterized in that In step 1, each image is token-encoded and then concatenated into a multimodal input token sequence. The specific steps include: First, each search area image and each initial template image are divided into non-overlapping or overlapping small blocks to obtain a fixed number of image blocks; Afterwards, each image block is mapped into a fixed-length token through linear projection. Each search area image and each initial template image consists of several tokens. Next, for the same modality, the tokens corresponding to the search area image and the initial template image are stacked sequentially to obtain a complete modality token sequence; Finally, the RGB modal Token sequence and the Depth modal Token sequence are concatenated in the Token dimension to form a fused multimodal input Token sequence.

3. The single-stream RGB-D target tracking method based on the state space model according to claim 1, characterized in that In step 1, the multimodal input token sequence is fed into a linear layer for feature embedding. The specific steps include: feeding the multimodal input token sequence into one or more linear layers, and performing feature dimensionality upscaling or downscaling to ensure that the token features meet the number of input channels required by the Mamba single-stream backbone network.

4. The single-stream RGB-D target tracking method based on the state space model according to claim 1, characterized in that The MSSM module performs the following calculations: ； ； in, is the RGB branch feature that has been preprocessed for the first time, is the Depth branch feature that has been preprocessed for the first time, SiLU is the activation function, Conv is the convolution layer, Linear is the linear layer, and Norm is the normalization operation; ； ； in, is the RGB branch feature that has been preprocessed for the second time; is the Depth branch feature that has been preprocessed for the second time; ； ； in, is the final mixed output of the RGB branch, is the final mixed output of the Depth branch, It is a multimodal state space module.

5. The single-stream RGB-D target tracking method based on the state space model according to claim 4 is characterized in that The state space unit establishes long-range dependencies through recursive state updates, specifically the update and output of the hidden state h, as follows: ； ； in, Hidden state parameters for this layer, Hide for the previous layer, is the input image feature; To output image features, A, B, and C are learnable parameters, and D is the connectivity matrix that provides features connecting input and output.

6. The single-stream RGB-D target tracking method based on the state-space model according to claim 1, characterized in that The training process uses a joint loss function to optimize the detection and positioning accuracy of the final target. The formula is as follows: ； in, is the cross entropy loss, is the L1 loss, is the GIoU loss, 、 、 The values ​​are 5, 1 and 2 respectively.

7. A single-stream RGB-D target tracking system based on a state-space model, characterized in that: include: Token generation and feature embedding module: Encodes RGB modality images and depth modality images into tokens respectively, concatenates them into a multimodal input token sequence, and embeds them into the feature space through a linear layer; Mamba single-stream backbone network: Multimodal input token sequences are input in a single stream. The Mamba single-stream backbone network uses a multi-layer Mamba structure for feature extraction and first-stage fusion. 3M module: performs the second stage fusion of the deep features output by the Mamba single-stream backbone network and decomposes them into RGB branch features through a linear layer and Depth branch features ,The MSSM module uses the RGB branch as the main input and the Depth branch as the auxiliary input to perform information interaction between the main and auxiliary modal features; Tracking head: performs multi-task prediction on the fused features and outputs classification heatmaps and bounding box prediction results; Training optimization module: uses a joint loss function to optimize the entire network to improve the accuracy of target detection and positioning.

8. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the single-stream RGB-D target tracking method based on the state space model according to any one of claims 1 to 6.

Images