RGB-T image saliency object detection method based on Mmba feedback iterative network

Abstract

The invention discloses an RGB-T image saliency object detection method based on a Mama feedback iterative network, and belongs to the field of computer vision. According to the network, a Mama encoder with double branches is adopted to extract multiple scales of features; then, a cross-layer feature fusion module is used for integrating features of a subsequent layer and features of a current layer, so that cross-scale correlation among multi-scale features extracted by Mama is enhanced, and significance performance of significant objects under different scales is enhanced; after cross-layer feature fusion, the feature enhancement module is then used for further extracting salient object information and increasing the proportion of salient features in a feature space; after the two modals are processed through the feature enhancement module to obtain refined features, the multi-modal feature fusion module combines the corresponding features in each layer to generate fused features, so that stronger semantics and details are shown; and finally, the generated features are sent to a feedback iteration architecture for two additional iterations to generate a clearer and more complete saliency map. The method is used for solving the common problems of feature detail loss, serious noise interference, poor physical consistency and the like of the saliency object detected in the prior art.

Description

Technical Field

[0001] This invention relates to the field of multimodal salient object detection technology in computer vision, specifically to a method for salient object detection in RGB-T images based on a Mamba feedback iterative network. Background Technology

[0002] In complex visual scenarios such as autonomous driving, security monitoring, and nighttime rescue, traditional salient object detection algorithms based on RGB images face severe performance bottlenecks. These methods rely on visual features such as texture and color in the visible light band for object discrimination. However, under low illumination, dynamic lighting changes, or complex background interference, the feature differences between the target and the background are significantly weakened, leading to false positives and false negatives in models based on shallow feature extraction. RGB-T images combine visible light texture information and thermal infrared temperature radiation information, effectively overcoming illumination limitations. Therefore, RGB-T salient object detection has become a research hotspot in the field of computer vision.

[0003] Current RGB-T salient object detection methods are mainly divided into two categories: Convolutional Neural Networks (CNNs) and Transformers. CNNs have strong local feature extraction capabilities, but are limited by their local receptive fields, making it difficult to model long-distance feature dependencies. Transformers can achieve global semantic associations, but suffer from high computational complexity and weak ability to capture local details. In recent years, the Mamba model in the State Space Model (SSM) has shown advantages in sequence and vision tasks due to its linear computational complexity and ability to model both global and local features, providing a new technical path for RGB-T salient object detection.

[0004] In traditional visual task processing, downsampling operations in the encoding / decoding architecture are a key factor leading to the degradation of spatial details of salient targets. Because existing Mamba-based methods lack dedicated feature feedback mechanisms designed for visual tasks, they struggle to compensate for lost details during downsampling, ultimately resulting in blurred boundaries and internal holes in the prediction results. Existing Mamba-based multimodal detection methods still suffer from the following shortcomings: First, in terms of multimodal fusion, the fusion methods of RGB and thermal infrared features generally have flaws. Most existing methods only use simple methods such as pixel-by-pixel addition or channel stitching, failing to establish cross-layer and cross-modal feature associations. This not only easily introduces noise and redundant information but also prevents the complementary advantages of the two modalities from being fully utilized. Second, from the perspective of loss function design, most methods rely too heavily on pixel-level supervision, neglecting the thermodynamic and physical properties of thermal infrared data. This design flaw makes it difficult for the model's predicted salient regions to match the thermal infrared radiation distribution, resulting in significantly insufficient generalization ability in complex thermal imaging scenarios. Third, regarding the balance between computational efficiency and modeling capability, while Transformer-based methods can effectively model global dependencies, their quadratic computational complexity makes them unsuitable for processing high-resolution RGB-T images. Furthermore, the traditional Mamba scanning mechanism still has significant room for improvement in terms of efficiency and accuracy of feature modeling because it does not consider the spatial continuity of visual targets. Therefore, designing an efficient Mamba-based RGB-T salient object detection architecture to achieve accurate feature extraction, multimodal deep fusion, and physical consistency constraints has important practical value and theoretical significance. Summary of the Invention

[0005] This invention provides a salient object detection method for RGB-T images based on the Mamba feedback iterative network, which solves the problems of loss of feature details, severe noise interference and poor physical consistency that are common in existing salient object detection methods.

[0006] This invention is achieved through the following technical solution: A method for salient object detection in RGB-T images based on a Mamba feedback iterative network is disclosed. The network employs a dual-branch Mamba encoder to extract features at multiple scales. A cross-layer feature fusion module then integrates features from subsequent layers with those from the current layer, thereby strengthening the cross-scale correlation between the multi-scale features extracted by Mamba and enhancing the saliency of salient objects at different scales. After cross-layer feature fusion, a feature enhancement module is used to further extract salient object information, increasing the proportion of salient features in the feature space. After the feature enhancement module processes the two modalities to obtain refined features, the multi-modal feature fusion module combines the corresponding features at each layer to generate fused features, exhibiting stronger semantics and details. Finally, the generated features are fed into the feedback iterative architecture for two additional iterations to generate a clearer and more complete saliency map.

[0007] Furthermore, the implementation process of the detection method is as follows: Step 1: Extract high-level features from the visual image Upsampled to the current layer visual image features Same spatial dimensions, then high-level features of the visual image With the visual image features of the current layer Stitching along the channel dimension; combining the visual image features of the current layer. Downsampling to high-level features of visual images Same spatial dimensions, then compared with high-level features of the visual image. The stitched fusion features are obtained by stitching along the channel dimension. Attention mechanisms are used to dynamically adjust feature weights and extract local contextual relationships between adjacent layers, resulting in refined features. ;Will and The fused features are obtained by concatenating along the channel dimension; finally, residual connections are introduced to combine the fused features with the original input features. Combined, the final cross-layer fusion features are obtained. ; Step 2: Combine the cross-layer fusion features obtained in Step 1 Features are fused across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Step 3: Perform multiplication, addition, and concatenation operations along the channels of the enhanced RGB-T features obtained in Step 2 to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; Step 4: Apply the features obtained in Step 3 The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

[0008] Furthermore, the formula in step 1 is defined as follows: (1) in, This indicates splicing along the channel dimension. and These represent upsampling and downsampling operations, respectively. This represents the channel attention function. This represents the spatial attention function.

[0009] Furthermore, the formula in step 2 is defined as follows: (2) in, This indicates an adaptive average pooling operation. This represents the convolution operation and the ReLU activation function; , This indicates a convolution operation.

[0010] Furthermore, the specific formula for step 3 is defined as follows: (3) in, This indicates the features of the significant region where RGB and thermal infrared characteristics overlap. This indicates that the original feature distributions of both modalities have been preserved. This indicates that features are concatenated along the channel dimension. This indicates that spatial attention is applied to the features. This indicates that channel attention is applied to the features. This indicates the convolution channel compression process.

[0011] Furthermore, step 4 specifically involves first defining a binary cross-entropy loss function. for: (4) in, Represents the coordinates of a pixel in the image. It is the first predicted by the decoder. i A significant graph, It is a truth value label; Define intersection-union ratio loss for: (5).

[0012] Furthermore, define SBL loss. for: (6) in, It is a thermal infrared image. The surface emissivity is set to 0.95. It is the Stefan-Boltzmann constant; TV loss was then defined. for: (7) in, The decoder predicts the first i The horizontal gradient of a saliency map, The decoder predicts the first i Vertical gradient of a saliency map, The smoothing coefficient is set to 0.001. It is the first predicted by the decoder. i The height of a saliency map, It is the first predicted by the decoder. i The width of a saliency map; Finally, the four types of losses described above are applied to the saliency maps of all four scales in the decoder output, and the summation is performed to obtain the final total loss. The formula is defined as: (8).

[0013] A system for salient object detection in RGB-T images based on a Mamba feedback iterative network, the system employing the aforementioned method for salient object detection in RGB-T images based on a Mamba feedback iterative network, the system comprising: Cross-layer fusion module: integrates high-level features of visual images Upsampled to the current layer visual image features Same spatial dimensions, then high-level features of the visual image With the visual image features of the current layer Stitching along the channel dimension; combining the visual image features of the current layer. Downsampling to high-level features of visual images Same spatial dimensions, then compared with high-level features of the visual image. The stitched fusion features are obtained by stitching along the channel dimension. Attention mechanisms are used to dynamically adjust feature weights and extract local contextual relationships between adjacent layers, resulting in refined features. Subsequently, the enhanced features from both directions are fused, integrating their complementary contextual information, and... and The fused features are obtained by concatenating along the channel dimension; finally, to refine the fused features and prevent gradient degradation during convolution operations, residual connections are introduced to connect the fused features with the original input features. Combined, the final cross-layer fusion features are obtained. ; Feature enhancement module: This module integrates the cross-layer fused features obtained from the cross-layer fusion module. Features are fused across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Multimodal feature fusion module: Performs multiplication, addition, and concatenation operations along the channels on the enhanced RGB-T features obtained from the feature enhancement module to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; Feedback Iterative Decoding Module: Combines the features obtained from the multimodal feature fusion module. The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

[0014] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0016] The beneficial effects of this invention are: The hybrid loss function proposed in this invention conforms to the physical laws of thermal imaging, which can achieve high-precision and robust salient object detection, and further improve the performance of RGB-T salient object detection.

[0017] This invention can model long-range dependencies with linear complexity, enabling reverse feature transfer between the decoder and encoder across multiple iterations, effectively eliminating redundant features and enhancing the performance of salient object detection. Attached Figure Description

[0018] Figure 1 This is a chart comparing the test metrics of the present invention with those of other models on the three datasets.

[0019] Figure 2 This is a comparison of PR curves on three datasets of this invention, where (a) is the result on the VT821 dataset, (b) is the result on the VT1000 dataset, and (c) is the result on the VT5000 dataset. Figure 3 This is a comparison chart of the predicted saliency maps of the present invention.

[0020] Figure 4 This is a flowchart of the method of the present invention.

[0021] Figure 5 This is a schematic diagram of the cross-layer feature fusion structure of the present invention.

[0022] Figure 6 This is a schematic diagram of the modal feature fusion module structure of the present invention.

[0023] Figure 7 This is a schematic diagram of the feedback iteration architecture of the present invention. Detailed Implementation

[0024] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0025] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] The following is in conjunction with the appendix to this application specification. Figure 1-7 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0029] Implementation Method 1 This embodiment provides a method for salient object detection in RGB-T images based on a Mamba feedback iterative network. The network of the detection method uses a Mamba encoder with two branches to extract features at multiple scales. Then, a cross-layer feature fusion module is used to integrate the features of the subsequent layers with the features of the current layer, thereby strengthening the cross-scale correlation between the multi-scale features extracted by Mamba and enhancing the saliency of salient objects at different scales. After cross-layer feature fusion, a feature enhancement module is then used to further extract salient object information, increasing the proportion of salient features in the feature space. After the feature enhancement module processes the two modalities to obtain refined features, the multi-modal feature fusion module combines the corresponding features at each layer to generate fused features, exhibiting stronger semantics and details. Finally, the generated features are fed into the feedback iterative architecture for two additional iterations to generate a clearer and more complete saliency map.

[0030] Furthermore, such as Figure 4 As shown, the implementation process of the detection method is as follows: Step 1: Extract high-level features from the visual image Upsampled to the current layer visual image features Same spatial dimensions, then high-level features of the visual image With the visual image features of the current layer Stitching along the channel dimension; combining the visual image features of the current layer. Downsampling to high-level features of visual images Same spatial dimensions, then compared with high-level features of the visual image. The stitched fusion features are obtained by stitching along the channel dimension. Attention mechanisms are used to dynamically adjust feature weights and extract local contextual relationships between adjacent layers, resulting in refined features. Subsequently, the enhanced features from both directions are fused, integrating their complementary contextual information, and... and The fused features are obtained by concatenating along the channel dimension; finally, to refine the fused features and prevent gradient degradation during convolution operations, residual connections are introduced to connect the fused features with the original input features. Combined, the final cross-layer fusion features are obtained. ; Step 2: Combine the cross-layer fusion features obtained in Step 1 When input to the feature enhancement module, features are fused from across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling to achieve spatial normalization; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Step 3: Perform multiplication, addition, and concatenation operations along the channels of the enhanced RGB-T features obtained in Step 2 to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; Step 4: Apply the features obtained in Step 3 The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

[0031] Furthermore, the formula in step 1 is defined as follows: (1) in, This indicates splicing along the channel dimension. and These represent upsampling and downsampling operations, respectively. This represents the channel attention function. This represents the spatial attention function.

[0032] Furthermore, the formula in step 2 is defined as follows: (2) in, This indicates an adaptive average pooling operation. This represents the convolution operation and the ReLU activation function; , This indicates a convolution operation.

[0033] Furthermore, the specific formula for step 3 is defined as follows: (3) in, This represents the intermediate features after splicing. The features primarily highlight the significant areas of overlap between RGB and thermal infrared features. This indicates that the original feature distributions of both modalities are preserved, which can prevent excessive feature suppression caused by multiplication operations. This indicates that features are concatenated along the channel dimension. This indicates that spatial attention is applied to the features. This indicates that channel attention is applied to the features. This indicates the convolution channel compression process.

[0034] Furthermore, step 4 specifically involves first constructing a basic detection loss to ensure that the predicted saliency map is highly consistent with the true annotations in terms of pixel classification and region overlap; then defining a binary cross-entropy loss function. for: (4) in, Represents the coordinates of a pixel in the image. It is the first predicted by the decoder. i A significant graph, It is a truth value label; Define intersection-union ratio loss for: (5).

[0035] Furthermore, the SBL loss is constructed, constraining the average radiant intensity of the predicted salient region to follow the same fourth-power relationship as the average temperature of the corresponding region in the thermal infrared imaging; the SBL loss is defined. for: (6) in, It is a thermal infrared image. The surface emissivity is set to 0.95. It is the Stefan-Boltzmann constant, and its value is It has no special physical meaning; its purpose is to prevent the denominator from being zero. Subsequently, a TV loss is constructed to suppress high-frequency noise and unnecessary detail fluctuations in the predicted saliency map, thereby making its boundaries smoother and more continuous; the TV loss is defined. for: (7) in, The decoder predicts the first i The horizontal gradient of a saliency map, The decoder predicts the first i Vertical gradient of a saliency map, The smoothing coefficient is set to 0.001. It is the first predicted by the decoder. i The height of a saliency map, It is the first predicted by the decoder. i The width of a saliency map; Finally, the four types of losses described above are applied to the saliency maps of all four scales in the decoder output, and the summation is performed to obtain the final total loss. The formula is defined as: (8).

[0036] This method was tested on three public datasets (VT821, VT1000, and VT5000). The VT821 dataset consists of 821 RGB images and their corresponding thermal infrared images, while the VT1000 and VT5000 datasets contain 1000 and 5000 multimodal images, respectively. In all experiments, half of the VT5000 dataset was used as training data, and VT821, VT1000, and the remaining VT5000 dataset were used as test data.

[0037] The network of this invention is compared with twelve state-of-the-art methods. For example... Figure 1As shown, these methods include MGAI, ACMANet, TIDNet, WaveNet, MCFNet, E2Net, LSNet, WGOFNet, ConTriNet, PATNet, PCNet, and FRPNet. To ensure fairness, all metrics for the compared methods were retested and generated using parameters provided by the authors.

[0038] To comprehensively evaluate the saliency performance of the network, eight evaluation metrics widely used in the field of RGB-T saliency target detection were introduced; these metrics include MAE , , , , and .

[0039] like Figure 1 The network shown achieves state-of-the-art saliency performance across most metrics on all three datasets. Even when the network of this invention does not rank first, its metrics generally remain within the top three. Regarding performance on each dataset, all methods were trained on the VT5000 dataset. The network of this invention achieves state-of-the-art results on half of the metrics on VT5000, while the remaining metrics, although not absolutely optimal, are very close to the highest values. For example, the maximum bias occurs... In the metrics, the difference only decreased from 0.917 to 0.912. These suboptimal metrics are attributed to the model's emphasis on overall performance and generalization ability, while PATNet prioritizes saliency performance on the training dataset (i.e., VT5000). Therefore, on the other two datasets (i.e., VT1000 and VT821), the network of this invention exhibits superior generalization ability, achieving best results in most metrics. In contrast, while PATNet demonstrates excellent saliency performance on VT5000, it performs only moderately on VT1000 and VT821. Meanwhile, WaveNet provides competitive metrics on VT1000 but performs poorly on the other two datasets. Overall, these comparisons confirm that the network of this invention outperforms competing methods in terms of saliency performance.

[0040] To verify the ability of the FIM-Net model of this invention to predict salient targets from the perspectives of precision and recall, PR curves were displayed on VT821, VT1000, and VT5000, as shown below. Figure 4 As shown, the red dashed line of FIM-Net forms the outer boundary of the dataset, demonstrating higher precision and recall. In VT5000, FIM-Net maintains higher precision than PATNet and PCNet while maintaining high recall (recall ≥ 0.85).

[0041] In VT821, the proposed FIM-Net model achieves higher recall than WaveNet and MGAI at very high accuracy (≥0.95), and in VT1000, its curve avoids early accuracy collapse compared to ACMANet and FRPNet. This is attributed to the feedback iteration mechanism and multimodal fusion reducing false positives while maintaining a high true positive rate.

[0042] At typical operating points, the balanced crossover point is closer to the top right corner than PATNet and WaveNet, the recall-first setting maintains accuracy compared to PCNet and FRPNet, and the precision-first setting maintains competitive recall compared to MGAI and ACMANet. These PR observations are consistent with... Figure 2 Consistent, among which FIM-Net's and It ranks highly.

[0043] In addition to the quantitative analysis experiments described above, the network of this invention was compared with ten methods on VT821 to predict saliency maps, where A represents a scene with small objects, B represents a scene with low illumination, C corresponds to a scene with multiple scale objects, and D represents a scene with low contrast.

[0044] like Figure 3 As shown, the feature fusion and enhancement mechanisms, along with the feedback iteration mechanism, successfully extracted sufficient features for the small object, enabling the output saliency map to clearly and completely display its precise contour. Conversely, when processing the input in the second row, methods such as MGAI and TIDNet incorrectly detect irrelevant background regions as salient objects. This phenomenon occurs because these methods lack dedicated modules for enhancing boundary features, resulting in insufficient ability to capture fine-grained boundary details of small objects. Therefore, without sufficient boundary features, they cannot accurately predict the contour and other detailed information of small objects. In low-light scenes, relying solely on RGB images makes it difficult for these methods to detect complete salient targets.

[0045] However, MCFNet benefits from its multimodal feature fusion module, which utilizes thermal infrared image features to achieve accurate saliency detection. While most methods perform well in this case, MCFNet exhibits incorrect outputs in the sixth row of input images. This is attributed to the fact that although MCFNet fuses thermal infrared image features, it does not fuse and enhance them with the corresponding RGB features, resulting in limited learning ability of thermal infrared features and thus poor saliency performance. For multi-scale target scenes, MCFNet effectively captures multi-scale features of salient objects through cross-layer feature fusion. In contrast, methods such as PCNet and E2Net lack effective mechanisms for extracting and enhancing multi-scale target features, causing them to miss some salient objects in the ninth row of input images. In low-contrast scenes, the hierarchical fusion operation in MCFNet's feedback iteration mechanism can accurately distinguish between salient objects and non-salient regions. However, some methods such as LSNet and ACMANet fail to accurately predict salient objects due to the lack of hierarchical processing of fused features.

[0046] 1) First, a dual-stream architecture is used to distinguish between the input RGB image and the thermal infrared image, and multi-scale features of RGB mode and thermal infrared mode are obtained. For features of different scales, cross-layer feature fusion is performed using channel attention and spatial attention mechanisms.

[0047] 2) After feature enhancement, the fused multi-scale cross-modal features are sent to the modal feature fusion module. Through complex addition and multiplication operations and attention mechanisms, adaptive complementary fusion between RGB images and thermal infrared features is achieved, and the fused cross-modal multi-scale features are output.

[0048] 3) After obtaining the fused cross-modal features, the feature enhancement module is used to refine the features and output the features for decoding or further iteration. After the features have been refined through a sufficient number of iterations, they will be input into the feature decoder to generate the final saliency map result.

[0049] Implementation Method 2 This embodiment provides a salient object detection system for RGB-T images based on a Mamba feedback iterative network. The detection system uses the salient object detection method for RGB-T images based on a Mamba feedback iterative network as described above. The detection system includes: Cross-layer feature fusion module: This module first upsamples high-level features and concatenates them with low-level features, and simultaneously downsamples low-level features and concatenates them with high-level features, thus constructing a bidirectional cross-scale feature association. Based on this, channel and spatial attention mechanisms are introduced to dynamically fuse and refine local contextual information between adjacent layers. Finally, by concatenating bidirectional enhanced features and adding residual connections, the module preserves the multi-scale characteristics of each layer while promoting the interaction of cross-layer contextual information, improving feature representation ability and gradient stability, enabling the network to more accurately detect and segment targets of different sizes. The specific workflow is as follows: Figure 5 As shown. The specific workflow is as follows: 1) First, the module performs bidirectional cross-scale feature association. By upsampling the high-level features and concatenating them with the low-level features along the channel dimension, and simultaneously downsampling the low-level features and concatenating them with the high-level features, fused features in two directions are obtained.

[0050] 2) An attention mechanism is introduced to dynamically refine the fused features. By sequentially applying channel attention and spatial attention functions, the module can adaptively adjust feature weights and effectively extract local contextual relationships between adjacent layers.

[0051] 3) Finally, complementary information is integrated and the final fused feature is output. The refined enhanced features in both directions are concatenated again along the channel dimension to aggregate into an aggregated feature containing multi-directional context. Residual connections are introduced to combine this aggregated feature with the original input feature, which effectively prevents gradient degradation while further refining the feature, ultimately outputting a more expressive and stable cross-layer fused feature.

[0052] Feature enhancement module: This module integrates the cross-layer fused features obtained from the cross-layer fusion module. When input to the feature enhancement module, features are fused from across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling to achieve spatial normalization; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Multimodal feature fusion module: Performs multiplication, addition, and concatenation operations along the channels on the enhanced RGB-T features obtained from the feature enhancement module to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; To address the limitations of single-modal processing in cross-layer and heavy-enhancement modules, this module effectively fuses enhancement features from both RGB and thermal infrared modalities. It establishes complementary and synergistic relationships between modalities through three basic interaction strategies: multiplication, addition, and concatenation. An attention mechanism is introduced to selectively enhance salient regions across modalities. Finally, channel compression generates fused features that balance semantic and detail information, thereby improving the model's robustness to salient targets, especially in low-contrast or complex background scenes. The detailed fusion process is as follows: Figure 6 As shown, the specific process is as follows: 1) By performing channel multiplication, addition, and splicing operations on the enhanced RGB and thermal infrared features respectively, features with different interactive properties are obtained.

[0053] 2) Apply spatial attention and channel attention to the features obtained by multiplication and addition respectively, and then concatenate the attention-enhanced features with the cascaded features along the channels to integrate complementary information from different interaction strategies.

[0054] 3) The spliced ​​high-dimensional features are compressed through convolution and the output features with rich semantics and spatial details are fused, thereby achieving effective complementarity and optimal fusion of RGB and thermal infrared modal information.

[0055] Feedback Iteration Module: Traditional encoder-decoder prediction networks, which employ single-round feature encoding and decoding, often fail to accurately predict the complete shape of salient targets due to missing key features and the presence of voids in the predicted salient targets. To address these issues, a feedback iteration structure based on visual state space is designed. When the decoder outputs the prediction result of the first layer, it is not considered the final output of the network. Instead, considering that it contains rich texture and detail information, it is used to enhance the features extracted by the Mamba encoder, thus achieving the purpose of feedback. The detailed enhancement process is as follows... Figure 7 As shown, the specific process is as follows: 1) First, the feature channel dimension of the encoder output is transformed into the last dimension, and a linear operation is performed to enhance the key features along the channel dimension. At the same time, in order to facilitate feature processing using convolution, the channel dimension of the last dimension of the features generated by the linear layer is restored to the second dimension, and its size is adjusted to match the original features.

[0056] 2) Secondly, one branch transforms features into trainable weights through operations such as pooling and linearization, while the other branch mainly models the long-range dependencies between boundary features and the output features of each encoder layer through a selective scanning mechanism to generate a feature matrix.

[0057] 3) Finally, the correlation between features is further enhanced using the feature matrix and trainable weights to highlight key features. Furthermore, the original output features of each encoder layer are introduced as residual information before fusion to prevent the loss of key features throughout the feature processing.

[0058] Feedback Iterative Decoding Module: Combines the features obtained from the multimodal feature fusion module. The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

[0059] Implementation Method 3 This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any step in Embodiment 1 by running the computer program stored in the memory.

[0060] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0061] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0062] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods described above can also be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0063] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0065] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments for mutual reference, and will not be repeated here.

[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0067] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for salient object detection in RGB-T images based on a Mamba feedback iterative network, characterized in that, The detection method employs a Mamba encoder with two branches to extract features at multiple scales. Then, a cross-layer feature fusion module is used to integrate the features of subsequent layers with the features of the current layer, thereby strengthening the cross-scale correlation between the multi-scale features extracted by Mamba and enhancing the saliency of significant objects at different scales. After cross-layer feature fusion, the feature enhancement module is then used to further extract salient object information and increase the proportion of salient features in the feature space. After the feature enhancement module processes the two modalities to obtain refined features, the multimodal feature fusion module combines the corresponding features at each layer to generate fused features, which exhibit stronger semantics and details. Finally, the generated features are fed into the feedback iterative architecture for two additional iterations to generate a clearer and more complete saliency map.

2. The saliency object detection method for RGB-T images according to claim 1, characterized in that, The implementation process of the detection method is as follows: Step 1: Extract high-level features from the visual image Upsampled to the current layer visual image features Same spatial dimensions, then high-level features of the visual image With the visual image features of the current layer Stitching along the channel dimension; combining the visual image features of the current layer. Downsampling to high-level features of visual images Same spatial dimensions, then compared with high-level features of the visual image. The stitched fusion features are obtained by stitching along the channel dimension. Attention mechanisms are used to dynamically adjust feature weights and extract local contextual relationships between adjacent layers, resulting in refined features. ;Will and The fused features are obtained by stitching along the channel dimension; Finally, residual connections are introduced to combine the fused features with the original input features. Combined, the final cross-layer fusion features are obtained. ; Step 2: Combine the cross-layer fusion features obtained in Step 1 Features are fused across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Step 3: Perform multiplication, addition, and concatenation operations along the channels of the enhanced RGB-T features obtained in Step 2 to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; Step 4: Apply the features obtained in Step 3 The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

3. The saliency object detection method for RGB-T images according to claim 2, characterized in that, The formula for step 1 is defined as follows: (1) in, This indicates splicing along the channel dimension. and These represent upsampling and downsampling operations, respectively. This represents the channel attention function. This represents the spatial attention function.

4. The saliency object detection method for RGB-T images according to claim 2, characterized in that, The formula for step 2 is defined as follows: (2) in, This indicates an adaptive average pooling operation. This represents the convolution operation and the ReLU activation function; , This indicates a convolution operation.

5. The saliency object detection method for RGB-T images according to claim 2, characterized in that, The specific formula for step 3 is defined as follows: (3) in, This indicates the features of the significant region where RGB and thermal infrared characteristics overlap. This indicates that the original feature distributions of both modalities have been preserved. This indicates that features are concatenated along the channel dimension. This indicates that spatial attention is applied to the features. This indicates that channel attention is applied to the features. This indicates the convolution channel compression process.

6. The saliency object detection method for RGB-T images according to claim 2, characterized in that, Step 4 specifically involves first defining the binary cross-entropy loss function. for: (4) in, Represents the coordinates of a pixel in the image. It is the first predicted by the decoder. i A significant graph, It is a truth value label; Define intersection-union ratio loss for: (5)。 7. The saliency object detection method for RGB-T images according to claim 6, characterized in that, Define SBL loss for: (6) in, It is a thermal infrared image. The surface emissivity is set to 0.

95. It is the Stefan-Boltzmann constant; TV loss was then defined. for: (7) in, The first predicted by the decoder i The horizontal gradient of a saliency map, The first predicted by the decoder i Vertical gradient of a saliency map, The smoothing coefficient is set to 0.

001. It is the first predicted by the decoder. i The height of a saliency map, It is the first predicted by the decoder. i The width of a saliency map; Finally, the four types of losses described above are applied to the saliency maps of all four scales in the decoder output, and the summation is performed to obtain the final total loss. The formula is defined as: (8)。 8. A saliency object detection system for RGB-T images based on a Mamba feedback iterative network, characterized in that, The detection system uses the RGB-T image salient object detection method based on the Mamba feedback iterative network as described in any one of claims 1-7, and the detection system includes: Cross-layer fusion module: integrates high-level features of visual images Upsampled to the current layer visual image features Same spatial dimensions, then high-level features of the visual image With the visual image features of the current layer Stitching along the channel dimension; combining the visual image features of the current layer. Downsampling to high-level features of visual images Same spatial dimensions, then compared with high-level features of the visual image. The stitched fusion features are obtained by stitching along the channel dimension. Attention mechanisms are used to dynamically adjust feature weights and extract local contextual relationships between adjacent layers, resulting in refined features. Subsequently, the enhanced features from both directions are fused, integrating their complementary contextual information, and... and The fused features are obtained by concatenating along the channel dimension; finally, to refine the fused features and prevent gradient degradation during convolution operations, residual connections are introduced to connect the fused features with the original input features. Combined, the final cross-layer fusion features are obtained. ; Feature enhancement module: This module integrates the cross-layer fused features obtained from the cross-layer fusion module. Features are fused across layers using depthwise separable convolution DSConv. Hierarchical features are extracted, followed by adaptive average pooling; finally, two convolutional layers are introduced after adaptive average pooling to filter out noise components, thereby further refining and enhancing the saliency of RGB-T features. Multimodal feature fusion module: Performs multiplication, addition, and concatenation operations along the channels on the enhanced RGB-T features obtained from the feature enhancement module to obtain the features. and Subsequently, an attention mechanism is used to selectively enhance salient regions while suppressing irrelevant regions, resulting in high-dimensional splicing features. Finally, the high-dimensional splicing features are... Using convolutional channel compression to generate output features ; Feedback Iterative Decoding Module: Combines the features obtained from the multimodal feature fusion module. The feature is further refined and fed into a feedback iterative architecture based on the visual state space for two additional iterations. Finally, the feature decoding after repeated iterations generates the final saliency map, and a hybrid loss function is introduced to train the decoder output. In addition to the binary cross-entropy (BCE) loss function and the intersection-over-union (IoU) loss function, the loss function also incorporates the Stefan-Boltzmann law (SBL) loss and the total variational TV loss.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images