Light-weight three-mode saliency target detection method based on Mama

Abstract

The invention discloses a lightweight three-mode saliency target detection method based on Mama, and the method comprises the steps: firstly, carrying out the preprocessing of a training set and a test set in a three-mode saliency target detection data set; secondly, constructing a lightweight three-mode salient target detection network based on Mamba, inputting test set data, and obtaining a detected target detection result; the three-mode saliency target detection network comprises a lightweight encoder based on Mama, a 3DMama fusion layer, and a decoder module based on a depth separable convolution DWConv. And finally, constructing a joint loss function to train a three-mode saliency target detection network, and testing the three-mode saliency target detection network. According to the method, complementary information is effectively fused, the efficiency and performance of the model are remarkably improved, and target detection is accurately and efficiently completed.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal information processing technology, specifically to a lightweight three-modal saliency target detection method based on Mamba. Background Technology

[0002] In modern computer vision applications, particularly in the field of salient object detection, deep learning models are typically used for feature extraction and classification. While existing salient object detection models have made significant progress in performance, these models usually have a large number of parameters and high computational complexity, making them difficult to run on resource-constrained edge devices, such as home service robots or industrial embedded systems. This necessitates the development of lightweight network architectures that can significantly reduce computational costs and energy consumption while maintaining high detection performance.

[0003] Most existing deep learning models employ CNN or Transformer architectures. However, both CNN and Transformer have their limitations in lightweight model design. While CNNs excel in fields such as image recognition, they primarily focus on local features, making it difficult to capture long-range dependencies. Furthermore, their computational efficiency decreases significantly with the length of the input sequence, hindering their ability to handle long input sequences. On the other hand, while Transformers can capture global information through self-attention mechanisms, their computational complexity increases quadratically with the input length, resulting in high computational costs when processing long sequences.

[0004] In recent years, trimodal salient object detection has gradually attracted attention. Trimodal fusion can represent objects more comprehensively; higher-quality images typically provide richer details and information, while lower-quality images may contain noise or lack certain features, which can lead to challenges in preserving detail and accuracy in the fused image. Existing multimodal salient object detection methods still face the following major problems:

[0005] 1. Quality Differences in Multimodal Fusion: Previous multimodal fusion methods, such as concatenation or element-level addition, only stack features without considering intermodal differences, easily introducing low-quality information and causing the fusion effect to be affected by inferior modalities. How to effectively handle the quality differences between modalities and avoid the negative impact of low-quality modalities on the fusion effect is one of the key issues in improving the performance of multimodal fusion.

[0006] 2. Large number of parameters and high computational complexity: In order to fully explore the potential of multimodal information, many models have introduced complex feature extraction and fusion mechanisms, which makes the model's parameter scale huge and significantly increases the computational complexity, making it difficult to deploy efficiently in resource-constrained environments.

[0007] 3. The Challenge of Efficiently Processing Long Input Sequences: Both CNN and Transformer architectures face efficiency bottlenecks when processing long input sequences. In practical applications such as video surveillance and robot vision, long input sequences are a common requirement. Designing efficient network architectures to achieve fast and accurate processing of long input sequences is a significant technical challenge in the field of salient object detection. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a lightweight Mamba-based method for trimodal salient target detection. This method balances detection performance and computational efficiency through a lightweight network structure, achieving high-precision trimodal salient target detection while effectively reducing the model's computational complexity and resource requirements. To achieve this goal, this invention proposes a lightweight Mamba-based encoder, a 3DMamba fusion layer, and a DWConv-based decoder module. Each module has a unique design and function to ensure accurate identification of salient targets.

[0009] A lightweight three-modal salient object detection method based on Mamba includes the following steps:

[0010] Step 1: Preprocess the training and test sets in the three-modal salient target detection dataset.

[0011] Preprocessing of the training set included resizing each image to the same size and employing various data augmentation strategies, including random flipping, rotation, and boundary clipping.

[0012] Preprocessing of the test set includes scaling and regularization of the test set data.

[0013] Step 2: Construct a lightweight trimodal salient object detection network based on Mamba, input the test set data, and obtain the target detection results.

[0014] Step 3: Train the trimodal saliency target detection network.

[0015] The preprocessed training set images are input into a trimodal salient object detection network, which outputs a predicted image of the same size as the input image. Next, the error between the predicted result and the ground truth label is calculated using a loss function, and the network weights are adjusted and optimized using backpropagation, combined with an optimizer and its parameter settings. After multiple rounds of training iterations, the desired network model is finally constructed.

[0016] Step 4: Testing. Input the preprocessed test set images into the trained network model to obtain the predicted images, compare them with the ground truth, and calculate various evaluation metrics.

[0017] Step 2 is as follows:

[0018] (1) Overall Network Architecture

[0019] This invention proposes a lightweight three-modal salient object detection network based on Mamba. Specifically, given an input image I = (I... v ,I d ,I t ), I v Represents a visible light image, I d Represents a depth image, I t These represent depth images, and all three are pixel-level images of the same scene; images Where C1, H, and W represent the number of channels, height, and width of the image, respectively. The trimodal image is first processed through a depthwise separable convolutional layer (DWConv) for shallow feature extraction, generating shallow feature maps. Subsequently, downsampling is performed using max-pooling.

[0020] Then, the features F of the three modes after downsampling are... D The inputs are fed into a four-layer lightweight encoder module based on Mamba, respectively, to obtain the encoder's multimodal features. Next, in the 3DMamba fusion layer, for the multimodal features obtained from the first three encoder layers, the multimodal features in the same layer are first stacked to form a new feature dimension, which is called the modality dimension. At this time, the stacked features... Stacked features are input into a 3D Mamba fusion layer, where global modeling and selective fusion of features are performed in the modality dimension to dynamically extract complementary information. This enables multimodal feature interaction to explore the complementarity between multimodal information. The final fused features are obtained by averaging the interacting multimodal features in the modality dimension. Specifically, for the multimodal high-level semantic features obtained from the fourth-layer encoder, the multimodal features are directly added element-wise to obtain the fused features. Finally, the fusion features will be used. The input to the first layer of the DWConv-based decoder module is used to progressively restore the feature size, and the corresponding fused features are added layer by layer from the second to the fourth layers of the DWConv decoder module. The features are added together with the output features of the previous layer and fused to obtain the salient target prediction map S.

[0021] The structure of each module will be explained in detail below.

[0022] (2) Lightweight encoder based on Mamba

[0023] This module consists of two key parts: the residual vision Mamba layer (RVM) and the DWConv layer used for downsampling.

[0024] Specifically, the encoder has three branches: an RGB branch, a depth (D) branch, and an infrared (T) branch. Taking the RGB branch as an example, the encoder module uses N1 consecutive RVM layers to capture global information, thereby reducing the number of parameters and making the entire model more lightweight; in the RVM layer, given the input RGB features... Initially, DWConv was used to reduce the number of feature channels by half, generating compressed features. Next, the spatial dimension H×W of the compressed features is flattened to obtain a sequence of length L=H·W, while keeping the channel dimension unchanged. Then, the sequence is transposed to obtain a two-dimensional sequence. Next, the transposed features Spatial long-range dependencies are captured using LayerNorm and the Visual State-Spatial Module (VSSM). Finally, the output of VSSM is compared with the compressed features. Perform residual summation and introduce a learnable adjustment factor. Control the residual ratio.

[0025] VSSM stands for Visual State-Spatial Module, in which features are stored... As input, the feature channels are fed into two parallel branches. In the first branch, the VSSM module expands the number of feature channels to λ×C using a linear layer, where λ represents a predefined channel expansion factor. The expanded features are then sequentially fed into DWConv, the SiLU activation function, the Spatial State Module (SSM), and a normalization layer to model long-range dependencies. In the second branch, the VSSM module expands the number of feature channels to λ×C using a linear layer, then uses the SiLU activation function. Subsequently, the VSSM module processes the features from both branches using element-wise multiplication and finally projects the number of channels back to C to generate a result similar to the input F. in Output F with the same shape out .

[0026] After processing by the VSSM module, the RVM layer uses another LayerNorm to adjust its output feature F. out Normalize it, and then use a projection layer to adjust its channel count to be the same as the input;

[0027] Finally, the feature map The sequence dimension L is decomposed back into H×W and transposed to generate features. The output characteristic F of the module is obtained by using a DWConv operation with a step size of 2. l+1 Its shape is

[0028] (3) DWConv-based decoder module

[0029] The decoder first uses addition to combine the fused features output by the 3D Mamba fusion layer on the same spatial scale. Features upsampled from the previous stage of the decoder Additive fusion is performed; subsequently, the feature maps are decoded using DWConv, residual connections, and the ReLU activation function; furthermore, an adjustment factor s is added to the residual connections. ′ To enhance decoding capabilities; the decoder ultimately uses bilinear interpolation to restore the prediction to the original resolution.

[0030] The beneficial effects of this invention are as follows:

[0031] 1. Reduced computational complexity and resource requirements: This invention introduces the Mamba model, which, through the Structured State Space (SSM) model, captures long-range dependencies of features while achieving near-linear time complexity. This effectively addresses the challenges of insufficient global information capture in CNNs and the high computational complexity of Transformers. Mamba also introduces a selection mechanism that can retain necessary and relevant data indefinitely while filtering out irrelevant information, significantly improving the model's efficiency and performance.

[0032] 2. Enhanced complementary fusion of multimodal information: A fusion module based on 3D Mamba is proposed, which stacks visible light, depth and thermal imaging features in the modal dimension. Through global perception and selective interaction mechanisms, low-quality modal interference is dynamically filtered out, and complementary information is effectively fused.

[0033] 3. Hardware-aware optimization: Optimized for GPU architecture, employing an efficient SRAM computing strategy to reduce data exchange between DRAM and SRAM, thus improving parallel computing efficiency. Combined with adjustment factor(s) and residual connections, it further accelerates model inference speed, making it suitable for applications with high real-time requirements. Attached Figure Description

[0034] Figure 1 This is a diagram of the overall network structure.

[0035] Figure 2 The diagram shows the residual vision Mamba layer and the visual state-space module structure.

[0036] Figure 3 This is a structural diagram of the 3DMamba fusion layer;

[0037] Figure 4 This is a comparison chart of the experimental results of this invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] A lightweight three-modal salient object detection method based on Mamba, such as Figure 1 As shown, the steps are as follows:

[0040] Step 1 is as follows:

[0041] The training and test sets in the trimodal salient object detection dataset are preprocessed separately.

[0042] Perform data augmentation and normalization on the training set. Perform size scaling and normalization on the test set.

[0043] Step 2 is as follows:

[0044] This invention proposes a lightweight three-modal salient object detection network based on Mamba. Specifically, given an input image I = (I... v ,I d ,I t ), I v Represents a visible light image, I d Represents a depth image, I t These represent depth images, and all three are pixel-level images of the same scene; images Where C, H, and W represent the number of channels, height, and width of the image, respectively; firstly, a depthwise separable convolutional layer (DWConv) is used for shallow feature extraction to generate shallow feature maps. Subsequently, downsampling is performed using max-pooling.

[0045] After that, as Figure 2 and Figure 3 As shown, the feature F of the three modes after downsampling D The inputs are fed into a four-layer lightweight encoder module based on Mamba, respectively, to obtain the encoder's multimodal features. Next, in the 3DMamba fusion layer, for the multimodal features obtained from the first three encoder layers, the multimodal features in the same layer are first stacked to form a new feature dimension, which is called the modality dimension. At this time, the stacked features... The stacked features are input into a 3D Mamba fusion layer to allow for sufficient multimodal feature interaction to explore the complementarity between multimodal information. The final fused features are obtained by averaging the interacting multimodal features along the modality dimension.

[0046] In particular, for the multimodal high-level semantic features obtained from the fourth-layer encoder, due to the large number of channels, in order to reduce the number of parameters, the multimodal features are directly added element-wise to obtain the fused features. Finally, the fusion features will be used. The input to the first layer of the DWConv-based decoder module is used to progressively restore the feature size, and the corresponding fused features are added layer by layer from the second to the fourth layers of the DWConv decoder module. The features are added together with the output features of the previous layer and fused to obtain the salient target prediction map S.

[0047] The structure of each module will be explained in detail below.

[0048] Lightweight encoder based on Mamba: This module consists of two key parts: residual vision Mamba layer (RVM) and DWConv layer used for downsampling.

[0049] Specifically, the encoder has three branches (RGB, D, T). Taking the RGB branch as an example, given a feature map... The encoder module first flattens and transposes the input feature map. Subsequently, the encoder module utilizes N1 consecutive RVM layers to capture global information; the RVM layers initially use DWConv to halve the number of channels. To reduce the number of parameters and make the entire model more lightweight, LayerNorm is used for layer normalization, followed by the Visual State-Spatial Module (VSSM) to capture spatial long-range dependencies. Then, adjustment factors are utilized. Achieving better performance through residual joins; this process can be expressed mathematically as follows:

[0050]

[0051] Where LN stands for LayerNorm; s represents the adjustment factor, which is a learnable parameter; + represents element-wise addition.

[0052] VSSM stands for Visual State-Spatial Module, in which features are stored... As input, it is fed into two parallel branches; in the first branch, the VSSM module uses a linear layer to expand the number of feature channels to λ×C, where λ represents a predefined channel expansion factor; subsequently, the features are fed sequentially into DWConv, the SiLU activation function, the spatial state module SSM, and a normalization layer to model the long-range dependencies of the features; the structure of the SSM layer combines the characteristics of convolutional neural networks and recurrent neural networks, treating the features at each spatial location as a time step in the sequence through linear state space modeling, and using discretized linear dynamic equations Ch t =e ΔA h t-1 +ΔBx t The input sequence is mapped to the output sequence. The learnable parameters A, B, and C retain the long-range memory of the recurrent network and can be computed in parallel using convolution or frequency domain FFT, achieving an approximately linear complexity of O(L log L). To further enhance content adaptability, A, B, and C are dynamically generated with the input, forming a selective state-space mechanism. This allows the model to maintain high efficiency while adaptively retaining or filtering information based on the importance of the current features, significantly enhancing its ability to capture global context and cross-modal complementary cues. In the second branch, the VSSM module uses a linear layer to expand the number of feature channels to λ×C, then uses the SiLU activation function. Subsequently, the VSSM module uses element-wise multiplication (Hadamard product, symbol ⊙) of features from both branches and finally projects the number of channels back to C to generate a sequence similar to the input F. in Output F with the same shape out The above process can be expressed by the following formula:

[0053] F1=LN(SSM(SiLU(DWConv(Linear(F in )))))

[0054] F2=SiLU(Linear(F in ))

[0055] F out =Linear(F1⊙F2)

[0056] After this, the RVM layer uses another LayerNorm to F out Normalization is performed, followed by adjustment using a projection layer. The number of channels is the same as the input; the above process can be described as:

[0057]

[0058] Next, the encoder module resizes and transposes the feature map to (C, HW), then performs a DWConv operation with a stride of 2 to reduce the resolution of the feature map and increase its dimensionality, resulting in the module's output feature F. l+1 Its shape is In particular, to improve computational efficiency, a downsampling operation, Max-Pooling, was performed before the first layer encoder, so the DWConv downsampling operation was not deployed in this layer encoder.

[0059] 3DMamba Fusion Layer: In the fusion stage, the features obtained from the three branches of the first three encoder layers are first combined. Stacking is performed along the modality dimension to form a stacked feature. The stacked features are then input into a 3D Mamba layer for feature interaction to explore the complementarity between the three modal features. Finally, the results are obtained through averaging operations along the modal dimensions. The features are then input into the decoder. Since the fourth-layer encoder has a large number of feature channels, to reduce computational load and parameters, the features obtained from the fourth-layer decoder are directly added element-wise to obtain the fused features. The input is fed into the decoder; the above process can be expressed by the following formula:

[0060]

[0061] Where ST represents the stacking dimension; Mean represents the average operation; This indicates element-wise addition.

[0062] Mamba represents the 3D Mamba operation, which first flattens the features after the dimensions are stacked. Then, through the Mamba layer, information is exchanged in three modalities to obtain... Finally, by adjusting the feature size and performing an averaging operation, we obtain... Information exchange between the three modalities was achieved through a 3D Mamba layer.

[0063] DWConv-based decoder module: for output from 3D Mamba fusion layer The decoder first performs feature fusion using addition; then, it decodes the feature maps using DWConv, residual connections, and the ReLU activation function; furthermore, an adjustment factor s is added to the residual connections. ‘ To enhance decoding capabilities; this process can be expressed mathematically as follows:

[0064] P out =ReLU(DWConv(Pin +F f )+s′·(P in +F f ))

[0065] The decoder ultimately uses bilinear interpolation to restore the prediction to the original resolution.

[0066] To address the issues of neglecting globality and diluting foreground pixel loss in SOD, and to ensure equal treatment of all pixels, the Pixel Position Aware Loss (PPA) function is adopted. This function improves upon the BCE loss (wBCE) and introduces a weighted IoU loss (wIoU), as shown in the following formula:

[0067]

[0068] Where 1(·) is the indicator function, γ is the hyperparameter, and the symbol l∈{0,1} represents two types of labels. and These are the predicted and actual values ​​of the pixel at position (i,j) in the image. This represents the predicted probability; in the loss function, each pixel is assigned a weight α, with complex pixels receiving a larger α and simple pixels receiving a smaller α; α can be considered an indicator of pixel importance, calculated based on the difference between the center pixel and its surrounding pixels, as shown in the following mathematical formula:

[0069]

[0070] Step 3 is as follows:

[0071] The preprocessed training images are fed into a three-modal salient object detection network for further processing, outputting a predicted image with the same size as the input image. Next, a loss function is used to quantify the difference between the predicted image and the ground truth annotations, yielding the loss value. Subsequently, the Adam optimizer is used to adjust the network parameters using backpropagation to reduce the loss value. During training, the batch size is set to 16, and the initial learning rate is set to 1×10⁻⁶. -4 Through multiple iterations and optimizations, a network model that meets the requirements was finally constructed.

[0072] The specific evaluation indicators in step 4 are as follows:

[0073] The evaluation metrics include precision and recall curves, F-Measure, mean absolute error (MAE), S-Measure, and E-Measure.

[0074] Precision and recall curves are standard metrics for evaluating model performance. Using thresholds from 0 to 255, the mean precision and mean recall values ​​for all 256 pairs across the saliency maps were fully computed.

[0075] The F-Measure is considered a comprehensive metric, obtained by weighted harmonic averaging of precision and recall.

[0076]

[0077] Where β 2 The hyperparameter was set to 0.3 to further emphasize the accuracy of salient target detection.

[0078] The absolute mean error (MAE) gives the absolute pixel-level error between the saliency mapping S and its corresponding ground truth GT, as shown in the following formula:

[0079]

[0080] Where W and H represent the width and height of the saliency plot, respectively.

[0081] The S-Measure measures the structural similarity of saliency maps, encompassing both region similarity (Sr) and object similarity (So). The definition is presented as:

[0082] S=α×S o +(1-α)×S r

[0083] The parameter α is set to 0.5.

[0084] E-Measure evaluates the similarity between the predicted saliency map and the ground truth by simultaneously combining local pixel saliency values ​​and image-level average saliency values:

[0085]

[0086] In the formula, f(·) is a convex function, and ° is the Hadamard product. Furthermore, in the bias matrix... and An alignment matrix ξ is constructed above, which is regarded as a center determination operation on the truth graph and the binary graph, respectively.

[0087] Network training setup:

[0088] This invention proposes a frequency domain decomposition and reconstruction network implemented using PyTorch on a PC equipped with an Intel(R) Core(TM) i9-12900K CPU and an RTX 3090 GPU. During training, the model's encoder is first pre-trained on the ImageNet dataset for 20 epochs, followed by fine-tuning of the overall model on the VDT-2048 dataset. Each image is resized to 352×352 and various data augmentation strategies are employed, including random flipping, rotation, and boundary clipping. Here, the Adam optimization network is used, with a batch size of 16 and an initial learning rate of 1×10⁻⁴, and the model is trained iteratively until convergence.

[0089] Furthermore, to fully verify the effectiveness of the model of this invention, extensive comparisons were conducted with existing multimodal salient object detection models on a publicly available trimodal salient object detection dataset, VDT-2048. VDT-2048 contains 2048 images and corresponding pixel-level annotations, of which 1048 images were used for training and 1000 images were used for testing. For fair comparison, the prediction results of all methods were provided by the respective authors or generated by running their source code, and the experimental data results are shown in Table 1.

[0090] Table 1 Quantitative Comparison Results of the Models

[0091]

[0092] As shown in Table 1, the model outperforms the existing lightweight VDT-SOD model MFDF-Net in all aspects, while reducing the model size and number of parameters by more than 40 times. While maintaining extremely low computational requirements, it further improves the F-measure and reduces the MAE, verifying the effectiveness of the lightweight design and proving that PPA-Loss effectively suppresses the problem of foreground pixel dilution. In addition, the higher S-measure and E-measure indicate that the network is superior to existing methods in terms of structural preservation and global consistency. On the RTX-3090, the measured FPS of 352×352 input can reach 65, which fully meets the requirements of real-time application scenarios, proving the effectiveness of our model.

[0093] from Figure 4 It can be seen that the model can capture the complementary relationship between the three modes very well, and can achieve accurate segmentation results based on depth and infrared image information even in dark or low light environments.

[0094] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0095] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A lightweight three-modal saliency target detection method based on Mamba, characterized in that, The steps include the following: Step 1: Preprocess the training and test sets in the three-modal salient target detection dataset, respectively; Step 2: Construct a lightweight trimodal salient object detection network based on Mamba, input the preprocessed test set data, and obtain the target detection results. Step 3: Construct a joint loss function to train a three-modal saliency target detection network, and test it using a preprocessed test set.

2. The lightweight three-modal saliency target detection method based on Mamba according to claim 1, characterized in that, The preprocessing is specifically implemented as follows: Preprocessing of the training set included: each image was resized to the same scale and data augmentation was performed using random flipping, rotation, and boundary clipping; Preprocessing of the test set includes scaling and regularization of the test set data.

3. The lightweight three-modal saliency target detection method based on Mamba according to claim 1, characterized in that, The lightweight three-modal salient object detection network based on Mamba is implemented as follows: Given an input image I = (I) in three modalities v ,I d ,I t ), I v Represents a visible light image, I d Represents a depth image, I t These represent depth images, and all three are pixel-level images of the same scene; images Where C1, H, and W represent the number of channels, height, and width of the image, respectively; the trimodal image is first processed through a depthwise separable convolutional layer (DWConv) for shallow feature extraction, generating shallow feature maps. Subsequently, downsampling was performed using max pooling. Then, the features F of the three modes after downsampling are... D The inputs are fed into a four-layer lightweight encoder module based on Mamba, respectively, to obtain the encoder's multimodal features. Next, in the 3DMamba fusion layer, for the multimodal features obtained from the first three encoder layers, the multimodal features in the same layer are first stacked to form a new feature dimension, which is called the modality dimension. At this time, the stacked features... Stacked features are input into a 3D Mamba fusion layer, where global modeling and selective fusion of features are performed in the modal dimension, and complementary information is dynamically extracted. Finally, the fused features are obtained by averaging the multimodal features after interaction in the modal dimension. For the multimodal high-level semantic features obtained from the fourth-layer encoder, the multimodal features are directly added element-wise to obtain the fused features. Finally, the fusion features will be used. The input to the first layer of the DWConv-based decoder module is used to progressively restore the feature size, and the corresponding fused features are added layer by layer from the second to the fourth layers of the DWConv decoder module. The features are added together with the output features of the previous layer and fused to obtain the salient target prediction map S.

4. The lightweight three-modal saliency target detection method based on Mamba according to claim 3, characterized in that, The lightweight encoder based on Mamba is implemented as follows: This lightweight Mamba-based encoder has three branches: an RGB branch, a depth (D) branch, and an infrared (T) branch. The three branches have identical structures. For one branch, the encoder module utilizes N1 consecutive Recurrent Virtual Machine (RVM) layers to capture global information. In the RVM layers, given the input RGB features... Initially, DWConv was used to reduce the number of feature channels by half, generating compressed features. Next, the spatial dimension H×W of the compressed features is flattened to obtain a sequence of length L=H×W, while keeping the channel dimension unchanged. Then, the sequence is transposed to obtain a two-dimensional sequence. Next, the transposed features Spatial long-range dependencies are captured using LayerNorm and the Visual State-Spatial Module (VSSM). Finally, the output of VSSM is compared with the compressed features. Perform residual summation and introduce a learnable adjustment factor. Control the residual ratio.

5. The lightweight three-modal saliency target detection method based on Mamba according to claim 4, characterized in that, The Visual State-Space Module (VSSM) is specifically implemented as follows: Features in this visual state-space module As input, it is fed into two parallel branches; in the first branch, the VSSM module uses a linear layer to expand the number of feature channels to λ×C, where λ represents a predefined channel expansion factor; then the expanded features are sequentially fed into DWConv, the SiLU activation function, the spatial state module SSM, and a layer-level modeling of the long-distance dependencies of the features; in the second branch, the VSSM module uses a linear layer to expand the number of feature channels to λ×C, and then uses the SiLU activation function; Subsequently, element-wise multiplication is used to process the features from the two branches, and finally the number of channels is projected back to C to generate a result similar to the input F. in Output F with the same shape out ; The RVM layer uses another LayerNorm to adjust the output feature F. out Normalize it, and then use a projection layer to adjust its channel count to be the same as the input; Finally, the feature map The sequence dimension L is decomposed back into H×W and transposed to generate features. A DWConv operation with a stride of 2 is used to reduce the resolution of the feature map and increase the dimensionality of the features, resulting in the output feature F of the module. l +1 Its shape is 6. The lightweight three-modal saliency target detection method based on Mamba according to claim 5, characterized in that, The DWConv-based decoder module is implemented as follows: The decoder first uses addition to combine the fused features output by the 3D Mamba fusion layer module on the same spatial scale. Features upsampled from the previous stage of the decoder Additive fusion is performed; then, sequential DWConv, residual connections, and ReLU activation functions are used to decode the feature maps; in addition, an adjustment factor s′ is added to the residual connections to enhance the decoding capability.

Images