Single-stream rgb-d object tracking method based on state space model

Abstract

The application discloses a single-flow RGB-D target tracking method based on a state space model, belongs to the technical field of computer vision, integrates RGB mode images and depth (Depth) mode image data in the form of Token, performs feature extraction and first stage fusion through an Mamba single-flow main stem network, then performs second stage fusion through a multi-modal Mix Mamba deep fusion module (3M module), improves the expression capability of cross-modal features and tracking efficiency, and finally outputs a prediction result by using a tracking head. Through the above mode, the application successfully solves the problems of large parameter quantity, low fusion efficiency and noise amplification in the traditional RGB-D target tracking method by means of Token-level fusion, an Mamba single-flow main stem network and an innovative 3M module, and significantly improves the efficiency and robustness of the tracker.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a single-stream RGB-D target tracking method based on a state space model. BACKGROUND

[0002] The single-target visual tracking task is to accurately track the position of the target in the subsequent frame under the condition that the target information in the first frame is given. Visual object tracking (VOT) is of great significance in the field of computer vision and is widely used in automatic driving, security monitoring and other fields. However, the traditional RGB target tracking has limited performance in the face of complex scenes (such as occlusion, extreme light). In recent years, scholars have tried to introduce other modalities (such as depth, thermal infrared) to make up for the shortcomings of the RGB modality and promote the development of RGB-D target tracking.

[0003] However, the existing RGB-D target tracking method still has problems such as large parameter quantity, inefficient fusion and noise amplification, and the dual-stream tracking framework based on Transformer not only has the problem of large training parameter quantity, but also has the practicality problem of slow inference speed. Therefore, it is necessary to design a simpler and more efficient modal fusion scheme, and the All Mamba framework can greatly reduce the training parameters and has a very high inference speed, which is an excellent choice for engineering practice.

[0004] Based on this, the present application designs a single-stream RGB-D target tracking method based on a state space model to solve the above problems. SUMMARY

[0005] In view of the above shortcomings of the prior art, the present application provides a simple, compact and efficient single-stream RGB-D target tracking method based on the All Mamba (All Mamba) architecture, called AMTrack. Through Token-level fusion, Mamba single-stream backbone network and innovative 3M module, the efficiency and robustness of cross-modal feature fusion are significantly improved, and it is suitable for various RGB-D target tracking scenes.

[0006] To achieve the above purpose, the present application realizes the following technical scheme:

[0007] A single-stream RGB-D target tracking method based on a state space model, comprising the following steps:

[0008] Step 1: For each video sequence, simultaneously acquire the RGB modality image and the depth modality image, and divide the images into search region images and initial template images respectively; Token encoding is performed on each image, and then the multi-modal input Token sequence is spliced and sent to a linear layer for feature embedding;

[0009] Step two: the multi-modal input Token sequence is input into the Mamba single-stream backbone network, first normalized, and then input into the backbone network composed of multiple stacked Mamba Blocks. In each Mamba Block, the multi-modal input Token sequence is input synchronously in a single-stream mode, and feature extraction and first-stage modal fusion are completed through the backbone network.

[0010] Step three: the features output by the backbone network are input into the 3M module, first normalized, and then decomposed into RGB branch features and Depth branch features by a linear layer; the RGB branch features are input into the MSSM module as the main input after convolution and activation function; the Depth branch features are directly reserved and input into the MSSM module as the auxiliary input after convolution and activation function.

[0011] Step four: the features output by the 3M module are input into the tracking head for target prediction.

[0012] Further, in step one, each image is Token encoded, and then spliced into a multi-modal input Token sequence. The specific steps include:

[0013] First, each search area image and each initial template image is divided into fixed number of image blocks with no overlap or small overlap.

[0014] Then, each image block is mapped to a fixed-length Token through linear projection, and each search area image and each initial template image is composed of several Tokens.

[0015] Next, for the same modality, the Tokens corresponding to the search area image and the initial template image are sequentially stacked to obtain a complete modal Token sequence.

[0016] Finally, the RGB modal Token sequence and the Depth modal Token sequence are spliced in the Token dimension to form a fused multi-modal input Token sequence.

[0017] Further, in step one, the multi-modal input Token sequence is input into a linear layer for feature embedding. The specific steps include: the multi-modal input Token sequence is input into one or more linear layers to increase or decrease the dimension of the Token features, so that the Token features meet the input channel number required by the Mamba single-stream backbone network.

[0018] Further, the MSSM module performs the following calculation:

[0019] ;

[0020] ;

[0021] wherein, is the first pre-processed RGB branch feature, is the first pre-processed Depth branch feature, SiLU is an activation function, Conv is a convolution layer, Linear is a linear layer, and Norm is a normalization operation;

[0022] ;

[0023] ;

[0024] wherein, is the second pre-processed RGB branch feature; is the second pre-processed Depth branch feature;

[0025] ;

[0026] ;

[0027] wherein, is the final mixed output of the RGB branch, is the final mixed output of the Depth branch, is a multi-modal state space module.

[0028] Further, the state space unit establishes long-distance dependency through recursive state update, specifically the update and output of the hidden state h, as follows:

[0029]

[0030]

[0031] wherein, is the hidden state parameter of this layer, is the hidden state of the previous layer, is the input image feature; is the output image feature, A, B, and C are learnable parameters, and D is a connection matrix providing connection between the input and output features.

[0032] Further, a joint loss function is used in the training process to optimize the detection and positioning accuracy of the final target, as follows:

[0033]

[0034] wherein, is the cross-entropy loss, L1 loss, GIoU loss, 、 、 respectively take 5, 1 and 2.

[0035] In order to better achieve the purpose of the application, the application also provides a single-flow RGB-D target tracking system based on a state space model, comprising:

[0036] Token generation and feature embedding module: encode the RGB modal image and the depth modal image into Tokens respectively, splice into a multi-modal input Token sequence, and embed into a feature space through a linear layer;

[0037] Mamba single-flow backbone network: the multi-modal input Token sequence is input in a single-flow mode, the Mamba single-flow backbone network adopts a multi-layer Mamba structure to perform feature extraction and first-stage fusion;

[0038] 3M module: the deep features output by the Mamba single-flow backbone network are subjected to second-stage fusion, and are decomposed into RGB branch features and Depth branch features by a linear layer, and the MSSM module performs information interaction between the main and auxiliary modal features with the RGB branch as the main input and the Depth branch as the auxiliary input;

[0039] Tracking head: multi-task prediction is performed on the fused features to output a classification heat map and a bounding box prediction result;

[0040] Training optimization module: a joint loss function is used to optimize the entire network to improve the accuracy of target detection and positioning.

[0041] In order to better achieve the purpose of the application, the application also provides a computer storage medium for storing program data, which, when executed by a computer, is used to implement the single-flow RGB-D target tracking method based on a state space model.

[0042] Compared with the prior art, the application has the following beneficial effects: 1. The Mamba single-flow backbone network adopts an efficient sequence modeling structure with linear complexity, can efficiently process large-scale Token sequences, has long-range dependency modeling capability, is suitable for visual tasks with strong spatial-temporal correlation such as target tracking, and realizes natural fusion of modal features from the bottom layer in a single-flow input mode, thereby avoiding information loss caused by early or late fusion, and greatly improving tracking efficiency and model compactness.

[0043] 2、3M module core MSSM module accepts the deep features of the main modalities (such as RGB) and auxiliary modalities (such as depth) as input, flexibly regulates and controls the contribution of different modal features to the hidden state and output result, and realizes efficient and non-saturated cross-modal fusion. The 3M module has low parameter quantity, is friendly to training, improves the cross-modal feature expression and fusion capability, significantly reduces the noise interference, and enhances the model robustness.

[0044] 3、The present application successfully solves the problems of large parameter quantity, inefficient fusion and noise amplification in the traditional RGB-D target tracking method through Token-level fusion, Mamba single-flow backbone network and innovative 3M module, and significantly improves the efficiency and robustness of the tracker.

[0045] 4、Experiments prove that the method of the present application performs well in multiple benchmark tests, is superior to the prior art in performance, speed and generalization capability, meets the requirements of real-time tracking, and has good universality and migratability. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0047] Figure 1 It is a framework diagram of the single-flow RGB-D target tracking method of the present application.

[0048] Figure 2 It is a schematic diagram of the paradigm difference between the present application and common target tracking methods. (a) is an existing double-flow tracking framework, (b) is an existing prompt learning tracking framework, (c) is an existing single-flow ViT tracking framework, and (d) is a single-flow All Mamba tracking framework of the present application.

[0049] Figure 3 It is a flowchart of the 3M module.

[0050] Figure 4 It is a visualization diagram of target classification scores after two-stage fusion of the present application. "#" in the diagram represents the frame number of the tested video.

[0051] Figure 5 It is a visualization of the tracking results of the present application and the existing algorithm Figure 1 (Adaptor_indoor scene).

[0052] Figure 6 It is a visualization of the tracking results of the present application and the existing algorithm Figure 2(Backpack Indoor Scene).

[0053] Figure 7 Bubble chart for performance-efficiency comparison of the present application and existing algorithms. Figure 3 (Duck Outdoor Scene).

[0054] Figure 8 Bubble chart for performance-efficiency comparison of the present application and existing algorithms. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] Embodiment One: Please refer to the accompanying drawings of the specification Figure 1 A single-flow RGB-D target tracking method based on a state space model, named AMTrack, includes the following steps:

[0057] Step One: Token Generation and Feature Embedding

[0058] 1.1, Multimodal image acquisition and preprocessing

[0059] For each video sequence, simultaneously obtain the RGB modality image and the depth (Depth) modality image. For the tracking task, the input image is divided into two types of regions:

[0060] Search Region (SR): From the current frame, according to the target position expansion of the previous frame by a certain proportion, the search region image is obtained, which contains the target and its surrounding environment, and ensures that the target will not be lost due to drift.

[0061] Initial Template (IT): The initial template image is the accurate cropping result of the target in the first frame or the initialization frame, which is used to provide a target reference for subsequent tracking.

[0062] For each frame, the search region image and the initial template image need to be extracted from the RGB modality and the depth modality image respectively, and finally four input image regions are obtained. All images are uniformly adjusted to a fixed size (such as 128x128), and standardized processing (mean-variance normalization, etc.) is performed to adapt to the input requirements of the subsequent Mamba single-flow backbone network.

[0063] 1.2, Token encoding and multimodal feature splicing

[0064] Firstly, each search region image and each initial template image is divided into fixed number of image patches without overlap or with small overlap (e.g. 16x16 Patch).

[0065] After that, for each image patch, a linear projection (e.g. fully connected layer) is used to map it into a fixed-length feature vector (Token). Each search region image and each initial template image is composed of several Tokens. At this point, the input image is encoded into a feature form of Token sequence, which is convenient for subsequent use.

[0066] Next, for the same modality (RGB or Depth), the Tokens corresponding to the search region image and the initial template image are respectively stacked in order to obtain the complete modality Token sequence.

[0067] Finally, the RGB modality Token sequence and the Depth modality Token sequence are spliced in the Token dimension to form a fused multi-modal input Token sequence.

[0068] 1.3, Feature Embedding

[0069] The multi-modal input Token sequence is input into one or more linear layers to increase or decrease the dimension of the Token feature, so that the Token feature meets the input channel number required by the Mamba single-stream backbone network. Further, position information encoding (Position Embedding) can be added to enhance the expression ability of the target position information.

[0070] Step two: Mamba single-stream backbone network for feature extraction and first-stage fusion

[0071] The multi-modal input Token sequence is input into the Mamba single-stream backbone network, which is first normalized (e.g. LayerNorm) to maintain numerical stability. Then it enters the backbone network composed of multiple layers (e.g. L layers) of Mamba Block to complete feature extraction and first-stage modality fusion.

[0072] In each Mamba Block, all multi-modal input Tokens (including RGB and Depth Tokens) are input synchronously in a single-stream manner, and the first-stage modality fusion and global context modeling (i.e. the process of establishing relationships between features) are completed simultaneously in the sequence modeling process (i.e. the image feature extraction process). Each Mamba Block (bi-directional Mamba encoder) contains the following main units:

[0073] Conv1d: a feature extraction unit composed of a simple convolution structure, which is used in different direction features and is divided into forward and backward Conv1d.

[0074] Forward SSM and backward SSM: both consist of core component state space module (SSM), where the forward SSM encodes the image Token forward as input when the feature is input, and the backward SSM encodes in reverse when encoding. Let the image feature have two feature arrangement directions when input.

[0075] State space module (SSM): core component, long-distance dependency is established by recursive state update, which is the update and output of hidden state h, using learnable parameters A, B, C, as follows:

[0076]

[0077]

[0078] wherein, is the hidden state parameter of this layer, is the hidden state of the previous layer, is the input image feature; is the output image feature, A, B, and C are learnable parameters, and D is a connection matrix providing connection between input and output features.

[0079] Linear projection layer: used to adjust the feature dimension.

[0080] Residual connection and layer normalization (LayerNorm): prevent gradient vanishing and accelerate training convergence.

[0081] Activation function (SiLU) and Dropout (regularization): improve non-linear expression ability and enhance model generalization.

[0082] The single independent output of each layer of the bidirectional Mamba encoder is .

[0083] Unlike traditional convolution or self-attention structure, Mamba single-flow backbone network can realize explicit information transmission between extremely long-distance Tokens through SSM mechanism, significantly enhancing the discrimination ability between target and background and across modalities. After multi-layer stacking, the feature sequence output by the Mamba single-flow backbone network contains rich spatial, contextual and preliminary cross-modal information, laying a foundation for subsequent deeper multi-modal fusion.

[0084] Mamba single-flow backbone network adopts an efficient sequence modeling structure with linear complexity, which can efficiently process large-scale Token sequences, has long-range dependency modeling capability suitable for spatial-temporal correlation strong visual tasks such as target tracking, and realizes natural fusion of modal features from the bottom layer, avoiding information loss caused by early or late fusion, greatly improving tracking efficiency and model compactness.

[0085] Step three: the multimodal Mix Mamba deep fusion module (3M module) performs second-stage fusion

[0086] To further enhance the cross-modal interaction of RGB and Depth features, the application designs a multimodal Mix Mamba deep fusion module with low parameter quantity and high efficiency for second-stage fusion after the backbone network.

[0087] As shown in the accompanying Figure 3 The features output by the L-layer backbone network are input into the 3M module, first normalized, and then decomposed into two independent features by a linear layer: RGB branch feature and Depth branch feature .

[0088] The RGB branch feature is sent to the MSSM module (Multimodal State Space Model) as the main input after convolution and activation function;

[0089] The Depth branch feature is directly retained for RGB self-modal completion and Depth cross-modal complement. After convolution and activation function, it is sent to the MSSM module as the auxiliary input.

[0090] The MSSM module takes the RGB branch as the main input and the Depth branch as the auxiliary input. The state transition matrix B and matrix C in the MSSM module are respectively associated with the auxiliary modal (Depth modal) feature, and the output is associated with the main modal (RGB modal) feature.

[0091] The MSSM module performs the following calculations:

[0092] ;

[0093] ;

[0094] wherein, is the first preprocessed RGB branch feature, is the first preprocessed Depth branch feature, SiLU is the activation function, Conv is the convolution layer, Linear is the linear layer, and Norm is the normalization operation;

[0095]

[0096]

[0097] wherein, is the second preprocessed RGB branch feature; The second pre-processed Depth branch feature;

[0098]

[0099]

[0100] wherein, is the final mixed output of the RGB branch, is the final mixed output of the Depth branch, is a multi-modal state space module;

[0101] The intermediate result after fusion output is multiplied by the RGB and Depth modal features respectively and then added to obtain the final fusion feature 、 , realizing efficient cross-modal information fusion and avoiding the noise amplification problem caused by traditional saturated interaction.

[0102] The MSSM module of the 3M module core accepts deep features of the main modal (such as RGB) and the auxiliary modal (such as depth) as input, flexibly regulates the contribution of different modal features to the hidden state and output result, and realizes efficient and non-saturated cross-modal fusion.

[0103] The 3M module has low parameter quantity and is friendly to training, improves the cross-modal feature expression and fusion ability, significantly reduces the noise mutual interference, and enhances the model robustness.

[0104] Step four: tracking head and joint loss function design

[0105] The features of the 3M module fusion output are input into the tracking head (Head). The tracking head is composed of multiple sub-convolution networks, which are respectively responsible for generating the classification score, the bounding box coordinates and the offset of the target, outputting the classification heat map and the bounding box prediction result, and realizing the position and category prediction of the target.

[0106] The training process adopts a joint loss function to optimize the detection and positioning accuracy of the final target, and the formula is as follows:

[0107]

[0108] wherein, is the cross-entropy loss, used to optimize the classification accuracy; is the L1 loss, used to optimize the regression accuracy of the bounding box; is the GIoU (Generalized Intersection over Union) loss, used to further improve the positioning robustness of the bounding box, 、 、 respectively taking values of 5, 1 and 2.

[0109] Embodiment two: a single-flow RGB-D target tracking system based on a state space model, comprising:

[0110] A token generation and feature embedding module: encoding the RGB modality image and the depth modality image into tokens respectively, splicing into a multi-modal input token sequence, and embedding into a feature space through a linear layer.

[0111] A Mamba single-flow backbone network: the multi-modal input token sequence is input in a single-flow mode, the Mamba single-flow backbone network adopts a multi-layer Mamba structure to perform feature extraction and first-stage fusion.

[0112] A 3M module: performing second-stage fusion on deep features output by the Mamba single-flow backbone network, specifically, decomposing the deep features into RGB branch features and depth branch features through a linear layer, and the MSSM module performs information interaction between main and auxiliary modal features by taking the RGB branch as the main input and the depth branch as the auxiliary input.

[0113] A tracking head: performing multi-task prediction on the fused features to output classification heat maps and bounding box prediction results, realizing position and category prediction of the target.

[0114] A training optimization module: optimizing the entire network using a joint loss function (cross-entropy, L1, GIoU) to improve the accuracy of target detection and positioning, and the formula is as follows:

[0115]

[0116] Further, AMTrack can flexibly adapt to different computing resource constraints and target sequence length scenarios by adjusting the depth of the 3M module or the token length. For specific application scenarios, different update intervals or fusion weights can be set to realize adaptive response to rapidly changing targets.

[0117] The present application successfully solves the problems of large parameter quantity, inefficient fusion and noise amplification in traditional RGB-D target tracking methods through token-level fusion, a Mamba single-flow backbone network and an innovative 3M module, significantly improving the efficiency and robustness of the tracker.

[0118] Embodiment three: a computer storage medium for storing program data, which, when executed by a computer, implements the single-flow RGB-D target tracking method based on a state space model of embodiment one.

[0119] Experimental example

[0120] The performance of the method (AMTrack) of the application is compared with other RGB and RGB-D target tracking methods, and the results are shown in Tables 1-2.

[0121] wherein Pr represents precision, indicating the proportion of actual positive samples in samples with a positive prediction result. Re represents recall, also known as the true positive rate, indicating the proportion of actual positive samples in the prediction result to the total number of positive samples. F-score is a weighted average of precision and recall, and the higher the F-score, the more robust the model. In addition, AMTrack-S represents that the backbone network adopts the Vim-S model, and AMTrack-M represents that the backbone network adopts the Vim-M model.

[0122] Table 1: Performance comparison results of the application method and other RGB and RGB-D target tracking methods on different test sets

[0123]

[0124] In the table: DAL-A Deep Depth-aware Long-term Tracker (deep depth-aware long-term tracker);

[0125] DeT-DepthTrack: Unveiling the Power of RGBD Tracking (DepthTrack: Unveiling the Power of RGBD Tracking);

[0126] SPT-RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking (a large-scale RGB-D target tracking dataset and benchmark);

[0127] ViPT-Visual Prompt Multi-Modal Tracking (Visual Prompt Multi-Modal Tracking);

[0128] VADT-Visual Adapt for RGBD Tracking (Visual Adapt for RGBD Tracking).

[0129] As can be seen from Table 1, the method of the application is significantly better than other RGB-D target tracking methods, indicating that the single-stream two-stage fusion model design is more robust for target tracking in complex scenes.

[0130] Table 2 Training amount, parameter amount and speed comparison of the method of the present application and other RGB and RGB-D target tracking methods on different test sets

[0131]

[0132] As can be seen from Table 2, the speed and parameter amount of the method of the present application are obviously superior to other RGB-D target tracking methods, and only need to be trained on a smaller and less data set to achieve a robust effect.

[0133] Figure 4 As can be seen: after the first stage fusion, the image feature score map can be well concentrated around the tracked target, and after the second stage fusion, the tracked target feature is more prominent.

[0134] Figures 5-7 As can be seen: the algorithm of the present application has higher tracking accuracy and is closer to the actual object frame.

[0135] The present application selects existing algorithms TABBTrack, UBPT, ViPT, DepthRefiner, DeT and SPT as a comparison, the above 6 algorithms belong to the field of RGB-D target tracking, and are published in authoritative journals or international conferences, which means that they have commonality in processing target tracking, which makes them become the recognized comparison benchmark. Figure 8 As can be seen: the red dashed line represents that the comprehensive performance and speed of the existing algorithm are lower than those of the present application. The algorithm of the present application not only has the highest EAO value, but also is higher than the remaining algorithms in terms of algorithm speed.

[0136] The above experimental results show that the method of the present application performs superiorly in multiple benchmark tests, and is superior to the prior art in performance, speed and generalization ability, meets the requirements of real-time tracking, and has good universality and migratability.

[0137] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A single-stream RGB-D object tracking method based on state space model, characterized in that, Comprising the following steps: Step one: for each frame of video sequence, simultaneously obtain RGB modality image and depth modality image, and respectively divide the images into search area image and initial template image; Token encoding is performed on each image, then the images are spliced into multi-modal input Token sequence, and the multi-modal input Token sequence is input into a linear layer for feature embedding; Step two: the multi-modal input Token sequence is input into the Mamba single-stream backbone network, first normalized, then input into the backbone network composed of multiple stacked Mamba blocks, in each Mamba block, the multi-modal input Token sequence is synchronously input in a single-stream mode, and the feature extraction and first stage modality fusion are completed through the backbone network; Step 3: The features output from the backbone network are input into the 3M module, where they are first normalized and then decomposed into RGB branch features through a linear layer. and Depth branch features RGB branch features After convolution and activation functions, the features are fed into the MSSM module as the main input; Depth branch features It is directly retained, and after passing through convolution and activation functions, it is fed into the MSSM module as auxiliary input; Step four: input the feature output by the 3M module into the tracking head for target prediction; The MSSM module performs the following calculation: ； = ； wherein, is the first pre-processed RGB branch feature, is the first pre-processed Depth branch feature, SiLU is an activation function, Conv is a convolution layer, Linear is a linear layer, and Norm is a normalization operation; ； ； wherein, is the second time pre-processed RGB branch feature; is the second time pre-processed Depth branch feature; ； ； wherein, is the final blended output for the RGB branch, is the final blended output for the Depth branch, is a multi-modal state space module.

2. The state space model based single-stream RGB-D object tracking method according to claim 1, wherein, In step one, Token encoding is performed on each image, then the images are spliced into multi-modal input Token sequence, the specific steps include: Firstly, each search area image and each initial template image is divided into fixed number of image blocks by non-overlapping or small overlapping blocks; Then, each image block is mapped to a fixed-length Token through linear projection, and each search area image and each initial template image is composed of several Tokens; Next, for the same modality, the Tokens corresponding to the search area image and the initial template image are respectively stacked in order to obtain the complete modality Token sequence; Finally, the RGB modality Token sequence and the Depth modality Token sequence are spliced in the Token dimension to form the fused multi-modal input Token sequence. 3.The state space model based monocular RGB-D object tracking method according to claim 1, wherein, In step one, the multi-modal input Token sequence is input into a linear layer for feature embedding, the specific steps include: the multi-modal input Token sequence is input into one or more linear layers, and the Token features are upgraded or downgraded to meet the input channel number required by the Mamba single-stream backbone network.

4. The state space model based single-stream RGB-D object tracking method according to claim 3, wherein, The state space unit establishes long-distance dependency through recursive state update, which is the update and output of the hidden state h, and the formula is as follows: ； ； wherein, is a hidden state parameter for this layer, is a hidden state for the previous layer, is an input image feature, is an output image feature, A, B, C are learnable parameters, is a connection matrix providing connections between input and output features.

5. The state space model based single-stream RGB-D object tracking method according to claim 1, wherein, The training process adopts a joint loss function to optimize the detection and positioning accuracy of the final target, and the formula is as follows: ； wherein, is a cross-entropy loss, is an L1 loss, is a GIoU loss, , , are respectively 5, 1 and 2.

6. A single-stream RGB-D object tracking system based on state-space models, characterized in that, Comprise: Token generation and feature embedding module: encode the RGB modality image and the depth modality image into Token respectively, splice into multi-modal input Token sequence, and embed into feature space through linear layer; Mamba single-stream backbone network: the multi-modal input Token sequence is input in a single-stream mode, and the Mamba single-stream backbone network adopts multiple Mamba structures for feature extraction and first stage fusion; 3M module: the second stage fusion of the deep features output by the Mamba single-stream backbone network, decomposed into RGB branch features and Depth branch features through a linear layer The MSSM module takes the RGB branch as the main input and the Depth branch as the auxiliary input to realize the information interaction between the main and auxiliary modal features.​ MSSM module performs the following calculation: ； = ； wherein is the first pre-processed RGB branch feature, is the first pre-processed Depth branch feature, SiLU is an activation function, Conv is a convolution layer, Linear is a linear layer, and Norm is a normalization operation. ； ； wherein, is the second time pre-processed RGB branch feature; is the second time pre-processed Depth branch feature; ； ； wherein, is the final blended output for the RGB branch, is the final blended output for the Depth branch, is a multi-modal state space module; Tracking head: multi-task prediction is performed on the fused features, and the classification heat map and bounding box prediction results are output; Training optimization module: the joint loss function is used to optimize the whole network to improve the accuracy of target detection and positioning.

7. A computer storage medium, characterized in that The computer storage medium is used for storing program data, which when executed by a computer, implements the single-stream RGB-D target tracking method based on a state space model according to any one of claims 1-5.

Images