General salient target detection method based on Mama

Abstract

The invention relates to the technical field of image processing and computer vision, in particular to a general salient target detection method based on Mama, which is characterized in that a salient guide Mama module is adopted in a decoder. The module comprises the following steps: acquiring a rough saliency map, after initializing scanning parameters, traversing image lines according to the current direction, and recording a saliency block index; the scanning direction is dynamically adjusted based on the proximity relation between the current saliency block and the next line, and it is ensured that space continuity is kept during feature serialization. The method comprises the following steps: acquiring multi-modal input such as RGB; extracting a plurality of modal features through a twinborn visual state space encoder; carrying out feature fusion through a multi-mode converter; and finally, inputting the RGB hierarchical features and the fusion features into a decoder containing a saliency guide Mamba module to generate a high-precision saliency prediction map. According to the method, the problem of insufficient global dependency relationship capture in traditional scanning is effectively solved, and the detection precision is improved while the calculation efficiency is kept.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a general salient object detection method based on Mamba. Background Art

[0002] Salient object detection is a crucial task in computer vision, aiming to identify and segment the most prominent and salient objects in an image. This task plays a key role in numerous applications, including object tracking, semantic segmentation, image enhancement, and autofocus. Currently, salient object detection relies primarily on two types of neural network architectures: convolutional neural networks (CNNs) and transformers. First, while CNNs can effectively extract features and exhibit good scalability through their hierarchical convolutional structures, their small receptive fields make it difficult to capture global dependencies, limiting their effectiveness in large-scale scenes. Second, while transformers can model global dependencies through their self-attention mechanism, providing superior visual modeling capabilities, their quadratic computational complexity results in low efficiency when processing large-scale images. Although numerous deep learning methods for salient object detection have been developed, the respective strengths and weaknesses of CNNs and transformers have limited their performance to some extent.

[0003] For example, Chinese patent CN118968019A discloses a small target detection method based on Mamba feature fusion. It proposes applying a Transformer-Mamba hybrid encoder to the feature fusion stage to simultaneously integrate shallow and deep semantic feature information, and combines it with a decoder stacking structure and a small target-specific loss function for detection optimization. This solution enhances the long-range dependency modeling capability through an attention mechanism and, to a certain extent, controls the number of model parameters. However, this method is primarily targeted at small target detection scenarios, and its core concept still relies on the Transformer's attention mechanism. It has not completely overcome its computational bottlenecks in multi-scale, high-resolution tasks, and lacks the ability to universally process multimodal input data (such as RGB images, depth maps, and heat maps). Summary of the Invention

[0004] This application addresses the difficulties of existing salient object detection methods, where CNNs struggle to model global dependencies and Transformers exhibit high computational complexity. By implementing a universal salient object detection method based on Mamba, this method addresses the core innovation of existing salient object detection methods. The method utilizes a saliency-guided Mamba algorithm module within the decoder, which achieves efficient feature extraction of salient regions through a unique spatial scanning strategy. This method can uniformly process multimodal data, including RGB images, depth images, and thermal images, to achieve accurate salient object detection results. During the feature extraction phase, a single-modality or twin encoder is automatically selected based on the input image modality. The hierarchical processing of the visual state space module enables global dependency modeling of multi-scale features, effectively capturing contextual information about salient objects. During the feature fusion phase, cross-modal interaction is achieved through selective state space blocks within the multimodal transformer, leveraging the complementary strengths of features from different modalities. During the decoding and prediction phase, the saliency-guided Mamba block maintains spatial continuity in salient regions, and combined with a context-aware upsampling method, high-precision alignment and fusion of multi-level features are achieved, achieving the technical effect of significantly improving detection accuracy while maintaining computational efficiency.

[0005] In a first aspect, embodiments of the present application provide a saliency-guided Mamba module algorithm, comprising the following steps: Input rough saliency map; Initialize the current scan row number to the starting row, set the initial scan direction to the first direction, and create an empty index storage sequence; Loop through the following until all rows have been processed: a. traverse the current row in the current scanning direction and record the saliency block indexes detected in the coarse saliency map in order to the storage sequence; b. If there is a salient block in the current row, then based on the position of the last salient block in the current row, calculate its relative position relationship with the nearest salient block in the next row; c. Dynamically adjust the scanning direction of the next row based on the comparison result of the relative position relationship: If the first condition is met, the scanning direction of the next row is set to the first direction; Otherwise, setting the scanning direction of the next row to a second direction opposite to the first direction; d. Increment the current line number to process the next line; The stored sequence is output, which contains all salient block indices arranged in the order of the scan path.

[0006] In the above implementation process, the saliency-guided mechanism effectively solves the problem of salient region fragmentation caused by traditional scanning methods. At the same time, combined with the efficient sequence processing capabilities of the Mamba model, it achieves a simultaneous improvement in the accuracy and computational efficiency of salient target detection.

[0007] In a second aspect, an embodiment of the present application provides a general salient object detection method based on Mamba, which may include: S1. Acquire an RGB image and other modal images of the image to be detected; the other modal images include a depth map and / or a thermal image; S2. Inputting the RGB image and the other modality image into the twin visual state space encoder to obtain multiple modality features; S3. Inputting the multiple modal features into a multimodal fusion converter, performing inter-modal interaction and outputting fusion features; S4. Obtain a rough saliency map from the hierarchical features of the RGB image in step S2 and the fused features in step S3 and input the map into a decoder. The decoder uses the saliency-guided Mamba module algorithm as described in claim 1 to obtain the saliency block index and generates a final saliency prediction map based on the saliency block index.

[0008] In some embodiments, the twin visual state space encoder includes four visual state space layers, each of which includes multiple visual state space modules, each of which performs the following operations: S211. Divide the image to be detected into original image blocks and perform layer normalization to obtain a normalized result; the normalized result includes a first data stream and a second data stream; S212. The first data stream is sequentially subjected to linear projection, reshaping, depthwise convolution, and SiLU activation and input into the SS2D module to model state space dependencies; The second data stream performs linear transformation and SiLU activation, and outputs a second processing result; S213. Multiply the first data result and the second data result and then perform a linear transformation output to obtain a linear layer processing result; S214. Add the original image block to the output result of the linear layer through a residual link to obtain multiple modal features; the multiple modal features include hierarchical features of the RGB image and deep features of other modal images, wherein the hierarchical features of the RGB image include three layers of shallow features and one layer of deep features.

[0009] In the above implementation process, by introducing a multi-layer visual state space module with state space modeling capabilities and using a residual connection structure to model image block features, the feature extraction effect at different scales is improved, and the expression ability of image context information is enhanced, which helps to accurately identify salient areas in complex scenes.

[0010] In some embodiments, the SS2D module modeling includes the following steps: S221. Scanning and expanding the two-dimensional feature map along four directions into a one-dimensional sequence; S222. Using selective state space blocks to model the state space dependencies between positions in a one-dimensional sequence; the state space dependencies include long-range dependencies; S223. Rearrange the processed one-dimensional sequence and sum it up.

[0011] In the above implementation process, by introducing a multi-directional scanning mechanism and a selective state space block into the SS2D module, a feature sequence expression structure combining local and global dependencies was constructed, which enhanced the modeling ability of spatial continuity and long-range dependencies in significant regions and improved the overall effect of feature modeling.

[0012] In some embodiments, the multimodal fusion converter in step S3 performs the following operations: S31. Performing linear transformation and depth convolution on the multimodal features to obtain a transformation result; S32. Flattening the transformation result into a two-dimensional tensor, concatenating the two-dimensional tensors to generate a joint feature sequence; S33. Inputting the joint feature sequence into a selective state space module, modeling the interaction between modalities and outputting the processed joint sequence; S34. Re-divide and linearly transform the processed joint sequence, output fusion features, and output fusion features.

[0013] In the above implementation process, by performing linear transformation, convolution, splicing and state-space interaction processing on multimodal features, the complementary characteristics between RGB modality and other modalities are effectively integrated, the accuracy of multi-source information fusion is improved, and the recognition ability of salient targets in multimodal scenarios is enhanced.

[0014] In some embodiments, when only the hierarchical features of the RGB image are input in step S31 , the multimodal fusion converter remains empty and transmits the hierarchical features of the RGB image to the decoder.

[0015] In the above implementation process, by skipping the converter module when only RGB images are input, the model can adaptively adjust its structure according to the task requirements, reducing redundant calculations and improving detection efficiency, while ensuring that the accuracy performance under single-modal input is not affected.

[0016] In some embodiments, a coarse saliency map is obtained based on the deep features in the multiple modal features, and the coarse saliency map is input into a saliency-guided Mamba module to generate enhanced features of the RGB image; the enhanced features include enhanced shallow features.

[0017] The enhanced shallow features and the fused features in step S3 are input into a visual state space decoder (VSS decorder layer) and multi-scale fusion is performed using a context-aware upsampling method. The context-aware upsampling method includes: S41. Performing linear transformation and depth convolution on the enhanced shallow features and the fusion features, respectively, to obtain the transformed enhanced shallow features and fusion features; S42. Pair the transformed enhanced shallow features and fusion features and perform up-sampling and down-sampling association, and output the up-sampling and down-sampling association results; S43. Serialize the up-sampling and down-sampling association results and input them into at least one selective state space module for processing, and output optimized hierarchical features.

[0018] In this implementation, the saliency-guided Mamba module enhances shallow features, enabling the network to more accurately focus on salient patches. This optimizes the support provided by shallow semantic features for the final prediction and improves the boundary clarity and structural integrity of salient patches. Furthermore, a context-aware upsampling method is used to align and fuse shallow enhanced features with fused features at multiple scales, significantly improving the spatial consistency and representation accuracy of feature integration at the decoder stage, effectively enhancing the clarity and stability of the final saliency prediction map.

[0019] In some embodiments, the method further includes a training step, in which supervised learning is used to optimize the neural network parameters. The loss function constructed during the training process includes a first binary cross-entropy loss between a rough saliency prediction map generated by deep features of the RGB image and the true label map, and a second binary cross-entropy loss between the final saliency prediction map and the true label map. The sum of the two losses is used as the total loss function for backpropagation.

[0020] In the above implementation process, by constructing dual-supervised learning and loss functions, the model is guided to focus on both the rough prediction and the saliency accuracy of the final output during the training phase, thereby improving the stability and convergence speed of the overall network training and achieving higher-quality salient object detection results.

[0021] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the above description when executing the computer program.

[0022] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the method described in the above description.

[0023] Compared with the prior art, the present invention has the following advantages: The present invention proposes a universal salient object detection method based on Mamba. The core innovation of the method lies in the adoption of a saliency-guided Mamba module algorithm.

[0024] First, the saliency-guided Mamba module algorithm uses an adaptive mechanism that dynamically adjusts the scanning direction. This mechanism calculates the relative position of salient regions in adjacent rows in real time and intelligently adjusts the scanning direction, ensuring the spatial continuity of salient regions during feature serialization. This effectively addresses the problem of salient target fragmentation caused by traditional scanning methods. While maintaining the linear computational complexity of the Mamba model, this algorithm optimizes the scanning path based on the saliency distribution, significantly improving feature representation quality and decoding efficiency. Its modality-independent design allows for seamless integration into multimodal data processing workflows, providing an efficient and reliable foundational module for salient target detection.

[0025] Overall, the present application proposes a general salient target detection method based on Mamba to address the problems of inconsistent structure, low fusion efficiency and difficulty in balancing computational efficiency and detection accuracy in existing salient target detection methods when processing multimodal inputs. A general detection method suitable for multiple modal data such as RGB images, depth maps, and thermal images is designed. In this method, by constructing a twin visual state space encoder, hierarchical feature extraction of RGB modalities and other modal images is realized, effectively retaining the semantic information and structural features of images of different modalities; further, interactive modeling and fusion of each modal feature is performed through a multimodal fusion converter, thereby improving the ability of information complementarity between modalities; finally, by fusing the hierarchical features of the RGB image and the fused features output by the converter, a decoder with a saliency-guided Mamba module algorithm is input to generate a prediction map with the ability to accurately locate salient areas. The technical solution of the present application effectively improves the generalization ability and detection accuracy of salient target detection in multimodal scenarios, and takes into account the reasoning efficiency while maintaining a clear model structure, and has good practicality and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of a general salient object detection method based on Mamba according to Example 1 of the present application; Figure 2 This is a schematic diagram of the network structure of a general salient object detection method based on Mamba in Example 2 of the present application; Figure 3 This is a schematic diagram of the structure of the Mamba module for saliency guidance in Example 2 of the present application; Figure 4 This is an algorithm flow chart of a saliency-guided Mamba module algorithm according to Example 2 of the present application; Figure 5 This is a schematic diagram of the context-aware upsampling method of Example 2 of the present application; Figure 6 This is a comparison chart of the results of the present method in Example 2 of the present application and the existing method. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments, as all technologies implemented based on the present invention fall within the scope of the present invention.

[0028] Example 1 During their research, the applicant discovered that when using convolutional neural networks (CNNs) for salient object detection, capturing global dependencies in an image to improve detection accuracy can only be achieved through stacking convolutional layers or dilated convolutions. However, this significantly increases computational complexity and makes it difficult to effectively model long-range dependencies. While the Transformer architecture can model global relationships through the self-attention mechanism, its quadratic computational complexity limits its application efficiency on large-scale images. When solving practical engineering problems, existing technologies struggle to balance global modeling capabilities and computational efficiency to achieve the technical goal of efficiently and accurately detecting salient objects in multiple scenarios. Therefore, after studying this issue, the applicant proposed a general salient object detection method based on Mamba. To address technical issues such as discontinuous feature representation in salient regions and inaccurate alignment of multi-level features, the applicant leverages a saliency-guided Mamba module and context-aware upsampling technology to maintain spatial continuity in salient regions and accurately align cross-level features, thereby achieving the technical effect of improving detection accuracy while maintaining linear computational complexity.

[0029] First, the core innovation of the Mamba-based general salient object detection method described in this application lies in the saliency-guided Mamba algorithm module used in the decoder. This module achieves efficient feature extraction of salient regions through an innovative spatial scanning strategy, specifically including the following key processing steps: Input rough saliency map; Initialize the current scan row number to the starting row, set the initial scan direction to the first direction, and create an empty index storage sequence; Loop through the following until all rows have been processed: a. traverse the current row in the current scanning direction and record the saliency block indexes detected in the coarse saliency map in order to the storage sequence; b. If there is a salient block in the current row, then based on the position of the last salient block in the current row, calculate its relative position relationship with the nearest salient block in the next row; c. Dynamically adjust the scanning direction of the next row based on the comparison result of the relative position relationship: If the first condition is met, the scanning direction of the next row is set to the first direction; Otherwise, setting the scanning direction of the next row to a second direction opposite to the first direction; d. Increment the current line number to process the next line; The stored sequence is output, containing all salient block indices arranged in the order of the scan path. This saliency guidance mechanism effectively solves the problem of salient region fragmentation caused by traditional scanning methods. Simultaneously, combined with the efficient sequence processing capabilities of the Mamba model, it achieves a simultaneous improvement in salient object detection accuracy and computational efficiency.

[0030] Please see Figure 1 , Figure 1 This is a flowchart of a general salient object detection method based on Mamba according to an embodiment of the present application. The method specifically includes: S1. Acquire an RGB image and other modal images of the image to be detected; the other modal images include a depth map and / or a thermal image; S2. Inputting the RGB image and the other modality image into the twin visual state space encoder to obtain multiple modality features; S3. Inputting the multiple modal features into a multimodal fusion converter, performing inter-modal interaction and outputting fusion features; S4. Obtain a rough saliency map from the hierarchical features of the RGB image in step S2 and the fused features in step S3 and input the map into a decoder. The decoder uses the saliency-guided Mamba module algorithm as described in claim 1 to obtain the saliency block index and generates a final saliency prediction map based on the saliency block index.

[0031] In some embodiments, the twin visual state space encoder includes four visual state space layers, each of which includes multiple visual state space modules, each of which performs the following operations: S211. Divide the image to be detected into original image blocks and perform layer normalization to obtain a normalized result; the normalized result includes a first data stream and a second data stream; S212. The first data stream is sequentially subjected to linear projection, reshaping, depthwise convolution, and SiLU activation and input into the SS2D module to model state space dependencies; The second data stream performs linear transformation and SiLU activation, and outputs a second processing result; S213. Multiply the first data result and the second data result and then perform a linear transformation output to obtain a linear layer processing result; S214. Add the original image block to the output result of the linear layer through a residual link to obtain multiple modal features; the multiple modal features include hierarchical features of the RGB image and deep features of other modal images, wherein the hierarchical features of the RGB image include three layers of shallow features and one layer of deep features.

[0032] In the above implementation process, by introducing a multi-layer visual state space module with state space modeling capabilities and using a residual connection structure to model image block features, the feature extraction effect at different scales is improved, and the expression ability of image context information is enhanced, which helps to accurately identify salient areas in complex scenes.

[0033] In some embodiments, the SS2D module modeling includes the following steps: S221. Scanning and expanding the two-dimensional feature map along four directions into a one-dimensional sequence; S222. Using selective state space blocks to model the state space dependencies between positions in a one-dimensional sequence; the state space dependencies include long-range dependencies; S223. Rearrange the processed one-dimensional sequence and sum it up.

[0034] In the above implementation process, by introducing a multi-directional scanning mechanism and a selective state space block into the SS2D module, a feature sequence expression structure combining local and global dependencies was constructed, which enhanced the modeling ability of spatial continuity and long-range dependencies in significant regions and improved the overall effect of feature modeling.

[0035] In some embodiments, the multimodal fusion converter in step S3 performs the following operations: S31. Performing linear transformation and depth convolution on the multimodal features to obtain a transformation result; S32. Flattening the transformation result into a two-dimensional tensor, concatenating the two-dimensional tensors to generate a joint feature sequence; S33. Inputting the joint feature sequence into a selective state space module, modeling the interaction between modalities and outputting the processed joint sequence; S34. Re-divide and linearly transform the processed joint sequence, output fusion features, and output fusion features.

[0036] In the above implementation process, by performing linear transformation, convolution, splicing and state-space interaction processing on multimodal features, the complementary characteristics between RGB modality and other modalities are effectively integrated, the accuracy of multi-source information fusion is improved, and the recognition ability of salient targets in multimodal scenarios is enhanced.

[0037] In some embodiments, when only the hierarchical features of the RGB image are input in step S31 , the multimodal fusion converter remains empty and transmits the hierarchical features of the RGB image to the decoder.

[0038] In the above implementation process, by skipping the converter module when only RGB images are input, the model can adaptively adjust its structure according to the task requirements, reducing redundant calculations and improving detection efficiency, while ensuring that the accuracy performance under single-modal input is not affected.

[0039] In some embodiments, a coarse saliency map is obtained based on the deep features in the multiple modal features, and the coarse saliency map is input into a saliency-guided Mamba module to generate enhanced features of the RGB image; the enhanced features include enhanced shallow features.

[0040] The enhanced shallow features and the fused features in step S3 are input into a visual state space decoder (VSS decorder layer) and multi-scale fusion is performed using a context-aware upsampling method. The context-aware upsampling method includes: S41. Performing linear transformation and depth convolution on the enhanced shallow features and the fusion features, respectively, to obtain the transformed enhanced shallow features and fusion features; S42. Pair the transformed enhanced shallow features and fusion features and perform up-sampling and down-sampling association, and output the up-sampling and down-sampling association results; S43. Serialize the up-sampling and down-sampling association results and input them into at least one selective state space module for processing, and output optimized hierarchical features.

[0041] In this implementation, the saliency-guided Mamba module enhances shallow features, enabling the network to more accurately focus on salient regions. This optimizes the support provided by shallow semantic features for the final prediction and improves the boundary clarity and structural integrity of salient regions. Furthermore, a context-aware upsampling method is used to align and fuse shallow enhanced features with fused features at multiple scales, significantly improving the spatial consistency and representation accuracy of feature integration at the decoder stage, effectively enhancing the clarity and stability of the final saliency prediction map.

[0042] In some embodiments, the method further includes a training step, in which supervised learning is used to optimize the neural network parameters. The loss function constructed during the training process includes a first binary cross-entropy loss between a rough saliency prediction map generated by deep features of the RGB image and the true label map, and a second binary cross-entropy loss between the final saliency prediction map and the true label map. The sum of the two losses is used as the total loss function for backpropagation.

[0043] In the above implementation process, by constructing dual-supervised learning and loss functions, the model is guided to focus on both the rough prediction and the saliency accuracy of the final output during the training phase, thereby improving the stability and convergence speed of the overall network training and achieving higher-quality salient object detection results.

[0044] The general salient target detection method based on Mamba provided in the embodiment of the present application can be applied to many technical fields, such as intelligent monitoring, autonomous driving, medical image analysis, etc., including specific application scenarios such as image segmentation, target tracking, and image enhancement. In the above implementation method, when performing salient target detection on multimodal image data, hierarchical features can be extracted through the twin visual state space encoder, cross-modal information interaction can be achieved through the multimodal fusion converter, and feature enhancement and fusion can be achieved through the saliency-guided scanning strategy and context-aware upsampling method, thereby achieving the effect of improving the accuracy and efficiency of salient target detection.

[0045] Example 2 As a further optimization of the previous embodiment, the embodiment of the present application provides a specific implementation of a general salient target detection method based on Mamba. The embodiment of the present application is based on the "encoder-decoder" structure (Encoder-DecoderStructure), and the encoder's RGB image and other modality image sub-networks are based on the visual state space layer network structure. Please refer to Figure 2 , Figure 2 This is a schematic diagram of the network structure of a general salient object detection method based on Mamba in an embodiment of the present application.

[0046] This method mainly includes the following steps: S1: Obtain RGB images, depth maps, thermal images, etc. of the image or video sequence to be detected.

[0047] S2: All input images are uniformly resized to 448×448 for training and testing.

[0048] When the task to be performed is the RGB salient object detection task (RGB SOD), in the embodiment of the present application, only the RGB image is input into the encoder to extract the hierarchical features of the RGB image. (in, ∈ 1, 2, 3, 4). When the task being performed is multimodal salient object detection, such as RGB-D SOD, RGB-T SOD, VSOD, or RGB-D VSOD, the RGB image is fed into the twin visual state space encoder along with images from other modalities (including depth maps, heat maps, or optical flow maps). In this application, the encoder contains four visual state space layers, each of which consists of multiple visual state space modules to extract image features at different scales.

[0049] The specific process of each visual state space module is as follows: for any modality, the input image will first be divided into multiple image blocks. These image blocks will be normalized and then divided into two information streams for feature modeling. The first information stream is processed by linear projection, reshaping operation, depth convolution and SiLU activation function in sequence, and then input into the SS2D module. The SS2D module is the "selective scanning two-dimensional module" and is used to model long-distance dependencies. The second information stream is directly fused with the first information stream after linear transformation and SiLU activation. Finally, the linear transformation is performed to complete the information integration, and it is added to the original input through residual connection to output the features.

[0050] The SS2D module's specific steps are as follows: The key process involves scanning two-dimensional image features in four different directions, expanding them into four one-dimensional sequences. The state-space dependencies between positions in these sequences are then calculated. This state-space dependency modeling encompasses not only the interactions between local features but also the global dependencies between long-range features, enabling a more comprehensive perception and modeling of salient regions. Sequences from different directions are then reordered and merged to complete information fusion.

[0051] Through the above steps, we can obtain the deep features of different modalities in the fourth visual state space layer. and RGB images in four visual state space layers. 1, 2, 3, 4 ） , where r represents the RGB modality, d represents the depth modality, f represents the optical flow modality, and t represents the thermal modality.

[0052] S3: The multimodal features obtained in step S2 (RGB mode) and (Other modalities, including depth modality, thermal modality, and optical flow modality) are input into the converter together to obtain the fused features When the task is RGB SOD, since the input image only contains RGB modalities, the converter module remains empty and the extracted RGB features are converted to Directly sent to the decoder. When the task is RGB-D / T SOD, the converter receives features from the RGB modality and other modal features , the feature sizes are , where H represents height, W represents width, C represents channel, and L is numerically equal to H W, that is, the original shape of H×W×C features is transformed into the shape of L×C. First, each modal feature is linearly transformed and deep convolution is performed to flatten its output features into a two-dimensional tensor with a shape of , then, the tensors from different modalities are concatenated along the L dimension to form a joint feature sequence. This sequence is then fed into a selective state space module to model the interaction dependencies between modalities. The processed sequence is then re-divided into two parts of the original modal structure output and , and then further integrated through addition and linear transformation operations to finally obtain the fusion feature Based on this dual-mode converter, it can be further expanded to support tri-modal input converter. It can be expressed as:

[0053] S4: The three shallow features extracted from the RGB image 1, 2, 3, 4) and by deep features Predicted rough saliency map The visual state space module used by this module in the processing process is the same as the feature This is different only in the SS2D module. Figure 3 , Figure 3 This is a schematic diagram of the structure of the saliency-guided Mamba module in the embodiment of the present application. Specifically, the SS2D module introduces the Spatial Neighboring Scanning (SNS) algorithm proposed in the embodiment of the present application. Based on the spatial continuity of the target area in the saliency map, it dynamically generates a sequence modeling path and expands the two-dimensional features into a structured one-dimensional sequence, thereby guiding the construction process of the spatial sequence and enhancing the modeling efficiency and saliency accuracy. Finally, the shallow features processed by this path are output as enhanced features. .

[0054] Please see Figure 4 , Figure 4The following is a flowchart of a saliency-guided Mamba module algorithm. The SNS algorithm in this module is used to extract salient regions from a two-dimensional image as a spatially continuous sequence input, thereby guiding the saliency-guided Mamba module to perform effective modeling. Specifically, the core steps of the SNS algorithm are: Algorithm input: A two-dimensional coarse saliency map Sc of shape (h, w); Initialization phase: Set the current scan line cut = 0, the initial scan direction dir is from left to right (l → r), and initialize the one-dimensional array Is to empty (for storing saliency block indices); Main loop process: Loop condition judgment: If cut ≤ h (the current line number does not exceed the total number of lines), continue execution; otherwise, output Is and end the algorithm; Current row scan: Scan the current row in the current direction dir and append the index of the scanned significant block to the array Is. If there is no significant block in the current row, jump directly to the row number increment step (SL); Distance calculation: Calculate the distance between the last salient block in Is and the leftmost salient block in the next row (dist_left) and the rightmost salient block (dist_right). If there is no salient block in the next row, jump directly to the row number increment step (SL). Direction decision: If dist_left ≤ dist_right, set the next line scan direction to left to right (l → r); otherwise set the next line scan direction to right to left (l ← r); Increment line number (SL): cut = cut + 1 (move to the next line); Return loop condition judgment Algorithm output: a one-dimensional array Is that stores the indices of all scanned salient blocks.

[0055] After obtaining the initial scan path Is for the salient blocks, to achieve structural modeling of the entire image, we further extract the indices of the non-salient blocks from the image and store them sequentially in a list Ins. This list is then concatenated with Is to construct the complete scan path for the 2D feature map. This concatenation process not only preserves the priority modeling order of the salient regions but also covers all regions of the image, providing the model with a complete foundation of contextual information during training.

[0056] Is is the algorithm's core intermediate variable, a one-dimensional sequence that essentially records the spatial index positions of all identified salient blocks in the coarse saliency map. Although structured as a list, Is is not a random collection but rather a "spatial path" that reflects the modeling sequence and determines the spatial access order for subsequent sequence modeling. Is, generated through a spatial proximity scanning strategy, accurately extracts contiguous salient regions while preserving their spatial structure, providing a well-structured and semantically focused input foundation for subsequent sequence modeling and feature enhancement.

[0057] In order to enhance the adaptability of the model to different spatial structures and target distributions, that is, the robustness of the SNS algorithm, this application designs three scanning path variants based on Is and Ins. First, retain the original order, put Is in front, and Ins in the back, forming an "Is+Ins" path; second, reverse Is and Ins respectively, put the reversed Ins in front, and the reversed Is in the back, forming a "reverse(Ins)+reverse(Is)" structure; third, reverse the two sequences in the same way, but retain the priority order of the saliency blocks, forming a "reverse(Is)+reverse(Ins)" path. These scanning paths are then applied to the RGB features, flattening them into a one-dimensional sequence, and finally, the input RGB features are enhanced into high-quality features. The remaining steps of SGMB (Saliency-Guided Mamba Block) are similar to those in the VSS block, guiding the Mamba module to model salient target regions from different sequence directions, thereby improving the robustness and generalization performance of the modeling.

[0058] S5: The enhanced three-layer shallow features obtained in S4 and the fused features output by the transformer The multi-layer feature upsampling adopts the context-aware upsampling method proposed in this application to improve the accuracy of feature fusion. The specific steps are as follows: S51. Input features: one is the fusion features from the deeper network , another enhanced feature from a shallower network In the first upsampling process, the two are and .

[0059] S52. Feature processing: Process the two input features separately, perform linear transformation and deep convolution operations respectively, and obtain the transformed features and .

[0060] S53. Feature Pairing and Context Modeling: The feature map in is divided into multiple groups according to the 2×2 window, and each group contains 4 feature blocks. Each feature block in is paired with these groups respectively, that is, through this pairing mechanism, the low-resolution features can be associated with their corresponding context information, so that they can be combined together to build a more accurate up- and down-sampling association. Figure 5 , Figure 5 A schematic diagram of a context-aware upsampling method according to an embodiment of the present application.

[0061] S54. After the combination is completed, the paired blocks are serialized and input into the selective state space module (S6 Block, S6 module): The paired blocks are combined into a long sequence and then input into the S6 module. By leveraging the causal prediction ability of the S6 module, the distribution pattern of low-resolution features can be gradually learned, and high-resolution features can be adjusted and aligned based on this distribution, thereby improving the accuracy of feature fusion.

[0062] S55. Rearrangement and Restoration: To further enhance context modeling, the context-aware upsampling method perturbs the connection order of paired sequences, generates a new order, and re-inputs it into another S6 module for further processing. This process restores the feature sequence to its original spatial structure, achieving higher alignment and fusion during integration, and improving the upsampling stage's ability to model the contextual relationship between high- and low-resolution features.

[0063] S6: Before using this network for salient object detection, it must be trained to optimize the neural network parameters. This training adopts a supervised learning approach, where the model is guided by labeled image data. During training, a loss function is constructed consisting of two parts: the first is the binary cross-entropy loss calculated between the coarse saliency prediction map generated by the deep RGB modality features extracted in step S2 and its corresponding true label map (i.e., the ground-truth map); the second is the binary cross-entropy loss between the final saliency prediction map and the ground-truth map. These two losses together constitute the total loss function during training, which is used to perform backpropagation and gradient updates on the model, thereby achieving gradual optimization of the network parameters.

[0064] In the embodiment of the present application, the gradient descent method is used to optimize the parameters during the model training process. The specific optimizer used is AdamW, and its initial learning rate is set to , the batch size is 2. After training is completed, the final model is obtained. Using the trained network, RGB images and other modality images are taken as input, and after the forward propagation process of the network, the final saliency detection results are output in the decoder part. Please refer to Figure 6 , Figure 6 The results of the present invention and the existing method are compared in Figure 2. Figure 6 In the figure, “Ours” is used to represent the proposed method, which is compared with the detection results of six existing methods and shows higher accuracy.

[0065] Example 3 As a further optimization of the preceding embodiment, an embodiment of the present application provides at least one computer device and at least one computer-readable storage medium for implementing a general salient object detection method based on Mamba provided in the present application.

[0066] Specifically, the computer device can be a general-purpose computing platform with image processing capabilities, such as a server or edge computing device including a graphics processing unit (GPU) and a deep learning inference acceleration module. The device integrates memory and a processor. The memory pre-stores a computer program. When the program is scheduled and executed by the processor, it executes the Mamba-based general salient object detection method described in this invention, which includes steps such as image preprocessing, feature extraction, multimodal fusion, saliency-guided enhancement, context-aware upsampling, and saliency map prediction. Deployment of this computer device enables rapid and accurate salient object detection in a variety of visual perception task scenarios.

[0067] The computer-readable storage medium can be in the form of a USB flash drive, solid-state drive, SD card, CD-ROM, embedded flash memory, or the like, storing a computer program for executing the above-described detection method. When this program is read and executed by a device with processing capabilities, it can achieve the same functional effects as the aforementioned computer device embodiment. This storage medium can be used for offline deployment or batch distribution of algorithm models, facilitating the rapid application and updating of the salient object detection system on different terminals.

[0068] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.

[0069] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A saliency-guided Mamba module algorithm, characterized in that The following steps are involved: Input rough saliency map; Initialize the current scan row number to the starting row, set the initial scan direction to the first direction, and create an empty index storage sequence; Loop through the following until all rows have been processed: a. traverse the current row in the current scanning direction and record the saliency block indexes detected in the coarse saliency map in order to the storage sequence; b. If there is a salient block in the current row, then based on the position of the last salient block in the current row, calculate its relative position relationship with the nearest salient block in the next row; c. Dynamically adjust the scanning direction of the next row based on the comparison result of the relative position relationship: If the first condition is met, the scanning direction of the next row is set to the first direction; Otherwise, setting the scanning direction of the next row to a second direction opposite to the first direction; d. Increment the current line number to process the next line; The stored sequence is output, which contains all salient block indices arranged in the order of the scan path.

2. A general salient object detection method based on Mamba, characterized in that: include: S1. Obtain the RGB image and other modality images of the image to be detected; The other modality images include depth images and / or thermal imaging images; S2. Inputting the RGB image and the other modality image into the twin visual state space encoder to obtain multiple modality features; S3. Inputting the multiple modal features into a multimodal fusion converter, performing inter-modal interaction and outputting fusion features; S4. Obtain a rough saliency map from the hierarchical features of the RGB image in step S2 and the fused features in step S3 and input the map into a decoder. The decoder uses the saliency-guided Mamba module algorithm as described in claim 1 to obtain the saliency block index and generates a final saliency prediction map based on the saliency block index.

3. The general salient object detection method based on Mamba according to claim 2, characterized in that: The twin visual state space encoder includes four visual state space layers, each of which includes multiple visual state space modules, and each visual state space module performs the following operations: S211. Divide the image to be detected into original image blocks and perform layer normalization to obtain a normalized result; the normalized result includes a first data stream and a second data stream; S212. The first data stream is sequentially subjected to linear projection, reshaping, depthwise convolution, and SiLU activation and input into the SS2D module to model state space dependencies; The second data stream performs linear transformation and SiLU activation, and outputs a second processing result; S213. Multiply the first data result and the second data result and then perform a linear transformation output to obtain a linear layer processing result; S214. Add the original image block to the output result of the linear layer through a residual link to obtain multiple modal features; the multiple modal features include hierarchical features of the RGB image and deep features of other modal images, wherein the hierarchical features of the RGB image include three layers of shallow features and one layer of deep features.

4. The general salient object detection method based on Mamba according to claim 3, characterized in that: The SS2D module modeling includes the following steps: S221. Scanning and expanding the two-dimensional feature map along four directions into a one-dimensional sequence; S222. Using selective state space blocks to model the state space dependencies between positions in a one-dimensional sequence; the state space dependencies include long-range dependencies; S223. Rearrange the processed one-dimensional sequence and sum it up.

5. The general salient object detection method based on Mamba according to claim 2, characterized in that: The multimodal fusion converter in step S3 performs the following operations: S31. Performing linear transformation and depth convolution on the multimodal features to obtain a transformation result; S32. Flattening the transformation result into a two-dimensional tensor, concatenating the two-dimensional tensors to generate a joint feature sequence; S33. Inputting the joint feature sequence into a selective state space module, modeling the interaction between modalities and outputting the processed joint sequence; S34. Re-divide and linearly transform the processed joint sequence, output fusion features, and output fusion features.

6. The general salient object detection method based on Mamba according to claim 5, characterized in that: When only the hierarchical features of the RGB image are input in step S31, the multimodal fusion converter does not perform any operation, and the hierarchical features of the RGB image are directly input to the decoder.

7. The general salient object detection method based on Mamba according to claim 6, characterized in that: Obtaining a coarse saliency map based on deep features among the multiple modal features, and inputting the coarse saliency map into a saliency-guided Mamba module to generate enhanced features of the RGB image; The enhanced features include enhanced shallow features; Input the enhanced shallow features and the fused features in step S3 into the visual state space decoder, and perform multi-scale fusion using a context-aware upsampling method; The context-aware upsampling method includes: S41. Performing linear transformation and depth convolution on the enhanced shallow features and the fusion features, respectively, to obtain the transformed enhanced shallow features and fusion features; S42. Pair the transformed enhanced shallow features and fusion features and perform up-sampling and down-sampling association, and output the up-sampling and down-sampling association results; S43. Serialize the up-sampling and down-sampling association results and input them into at least one selective state space module for processing, and output optimized hierarchical features.

8. The general salient object detection method based on Mamba according to claim 7, characterized in that: The method also includes a training step, in which a supervised learning method is used to optimize the neural network parameters. The loss function constructed during the training process includes a first binary cross-entropy loss between a rough saliency prediction map generated by deep features of the RGB image and the true label map, and a second binary cross-entropy loss between the final saliency prediction map and the true label map. The sum of the two losses is used as the total loss function for backpropagation.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images