RGB-D saliency target detection method

Abstract

The invention discloses an RGB-D saliency target detection method, and relates to the technical field of computer vision. Comprising the following steps: acquiring a color image, depth information and a corresponding RGB-D saliency target annotation graph from an RGB-D saliency detection data set; inputting the color image and the depth information into a cross-modal saliency detection network to obtain an RGB-D saliency target prediction map; the cross-modal saliency detection network is trained through the RGB-D saliency target prediction map and the RGB-D saliency target annotation map, and the trained cross-modal saliency detection network is obtained; and inputting a to-be-processed color image and to-be-processed depth information into the trained cross-modal saliency detection network to obtain an RGB-D saliency target recognition graph. According to the method, the visual integrity and detail fidelity of saliency detection are remarkably improved, and the accuracy of a saliency detection result is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an RGB-D salient object detection method. Background Art

[0002] Salient object detection (SOD) is a core task in computer vision, aiming to simulate the human visual attention mechanism to accurately identify the most visually attractive objects or regions in complex environments. With the development of deep learning, saliency detection has become a research hotspot due to the rapid development of multimodal perception technologies, attracting widespread attention. Compared with traditional RGB single-modality methods, RGB-D salient object detection (RGB-D SOD) aims to leverage the complementarity between RGB images and depth information to improve the detection accuracy of salient objects. RGB-D saliency detection utilizes depth information to compensate for the limitations of the visible light modality in complex scenes. It is widely used in the preprocessing steps of various complex visual tasks, showing great potential for development in application areas such as autonomous driving, robot navigation, and augmented reality.

[0003] With the introduction of the Transformer mechanism, research in this field has gradually evolved from relying solely on convolutional neural networks (CNNs) to a hybrid architecture that combines CNNs and Transformers. However, regardless of whether the network backbone uses CNNs or Transformers, most RGB-D SOD methods still use the encoder-decoder paradigm for feature processing and reconstruction. Currently, in RGB-D SOD tasks, encoder-decoder architectures can be mainly summarized into three types: (1) directly connecting the encoder through a feature fusion module so that dual-stream features are passed from the encoder to the decoder (e.g., CDINet); (2) first integrating the dual-stream encoded features in the fusion module and then inputting them into the decoder in a single-stream form (e.g., CCAFNet); (3) first completing feature interaction processing in the fusion module and then transmitting the fused dual-stream features to the decoder (e.g., HCT). In this task, the main role of the encoder is to extract features with a large global receptive field and rich semantic information from the input image in order to more effectively capture salient objects and their structures. The decoder's responsibility is to map these features back to the spatial distribution of the input image to restore edge details and local information. Since the features generated by the decoder are usually closer to the original image, fusing the high-level representations from the encoding stage with the detailed information from the decoding stage can further improve the overall detection performance of RGB-D SOD.

[0004] Several major challenges remain in current research. First, depth images and their extracted features often suffer from poor quality, for example, due to variations in illumination, complex backgrounds, or sensor errors, and may contain significant noise. Direct fusion with RGB features can lead to noise propagation, weakening the model's discriminative capabilities and thus increasing the difficulty of saliency prediction. Therefore, effectively assessing the reliability of depth information and designing robust fusion mechanisms remain pressing challenges. Second, efficiently fusing cross-modal features is a key issue in RGB-D SOD research. The RGB modality primarily focuses on surface features such as color and texture, while the depth modality provides spatial structural information. Given the significant differences in the distribution of these two modalities in feature space, designing appropriate fusion strategies that leverage their complementary strengths has become a research priority. Furthermore, balancing global saliency inference with the preservation of local detail is equally crucial. While depth information offers advantages in structural description, its low spatial resolution can lead to loss of detail.

[0005] In summary, although existing research methods have explored multi-scale feature modeling, they have limitations in simultaneously taking into account global saliency reasoning and local boundary optimization, which will affect the accuracy of saliency detection results. Summary of the Invention

[0006] Based on this, it is necessary to provide an RGB-D salient object detection method to address the above technical problems.

[0007] An embodiment of the present invention provides an RGB-D salient object detection method, comprising: Obtain color images, depth information, and corresponding RGB-D salient object annotations from the RGB-D saliency detection dataset; The color image and depth information are input into the cross-modal saliency detection network to obtain an RGB-D salient object prediction map; the cross-modal saliency detection network is trained with the goal of minimizing the deviation between the RGB-D salient object prediction map and the RGB-D salient object annotation map to obtain a trained cross-modal saliency detection network; The color image to be processed and the depth information to be processed are input into a trained cross-modal saliency detection network. The trained cross-modal saliency detection network includes: a parallel encoder, a cross-modal calibration module, a cross-modal attention enhancement module including a first interactive branch and a second interactive branch, a dual attention fusion module and a decoder connected in sequence; the parallel decoder includes an RGB encoder and a depth encoder, and an edge refinement perception module is added between the input end of the RGB encoder and the output end of the decoder; the RGB features of the color image are extracted by the RGB encoder, and the depth features of the depth information are extracted by the depth encoder; the RGB features and the depth features are transformed and spliced ​​by the cross-modal calibration module to obtain an initial fused feature; the initial fused RGB features are weight-normalized by the first interactive branch to obtain an RGB feature stream; the initial fused features and RGB features are weight-normalized by the second interactive branch to obtain a depth feature stream; the RGB feature stream and the depth feature stream are spliced ​​by the dual attention fusion module to obtain a final fused feature; the final fused feature is decoded by the decoder to obtain a final decoded feature; the color image and the final decoded feature are processed by the edge refinement perception module to obtain an RGB-D salient object recognition map.

[0008] Optionally, the RGB features and depth features are transformed and concatenated through a cross-modal calibration module to obtain initial fusion features, specifically including: The RGB features and depth features are transformed through the fully connected layer based on the following formula: ; in, is the RGB feature, is the deep feature, is the RGB feature after feature transformation, is the depth feature after feature transformation; The deep features after feature transformation are enhanced through the channel attention mechanism and the spatial attention mechanism based on the following formula: ; in is the deep feature processed by the channel attention mechanism, is the enhanced deep feature, LinearLayer is the fully connected layer, CA is the spatial attention mechanism, and SA is the channel attention mechanism; The enhanced depth features and RGB features are spliced ​​together through the following splicing operation to obtain the splicing result: ; Among them, C is the splicing operation, is the splicing result; The concatenation result and the deep features are residually connected based on the following formula to obtain the initial fusion features: ; in, CBR Represents Conv+BN+ReLU operation, are learnable weights.

[0009] Optionally, weight normalization is performed on the initial fusion features through the first interaction branch to obtain an RGB feature stream; and weight normalization is performed on the initial fusion features and the RGB features through the second interaction branch to obtain a depth feature stream, specifically including: The initial fusion features and RGB features are upsampled and preliminarily convolved based on the following formula: ; in, To decode the features, For RGB features Decoded RGB high-level features, is the initial fusion feature Decoded deep high-level features, U is the upsampling operation, S is the Sigmoid activation function; The initial fusion feature is processed by the first interactive branch to obtain an enhanced initial fusion feature; the RGB feature is processed by the second interactive branch to obtain an enhanced RGB feature; The enhanced initial fusion features and enhanced RGB features are mapped to query, key and value respectively. The attention weights are determined and the RGB feature stream and the depth feature stream are generated through matrix multiplication and Softmax function based on the following formula: ; in, is the RGB feature stream, is the deep feature flow.

[0010] Optionally, the enhanced initial fusion features and enhanced RGB features are mapped to query, key, and value, respectively, based on the following formula: ; in, To enhance the initial fusion features, To enhance RGB features, To enhance the query of RGB features, To enhance the RGB feature key, To enhance the value of RGB features, To enhance the query of the initial fusion features, To enhance the key of the initial fusion feature, To enhance the value of the initial fusion feature, To enhance the query weight of RGB features, To enhance the key weight of RGB features, To enhance the value weight of RGB features, To enhance the query weight of the initial fusion features, To enhance the key weights of the initial fusion features, is the value weight to enhance the initial fusion feature.

[0011] Optionally, the RGB feature stream and the depth feature stream are spliced ​​together through a dual attention fusion module to obtain the final fused features, specifically including: Perform average pooling operations on the RGB feature stream and the depth feature stream to extract global context information; The pooled features are concatenated and transformed through convolution blocks to obtain intermediate fusion features; Perform element-wise multiplication of the intermediate fusion features with the RGB feature stream and the depth feature stream respectively, and add them to the original features to obtain preliminary enhanced RGB features and preliminary enhanced depth features; Enhance the RGB features and depth features through channel attention and spatial attention to obtain attention-enhanced RGB features and attention-enhanced depth features; The RGB features after attention enhancement are adaptively selected through the Sigmoid function, and multiplied and added with the preliminary enhanced RGB features to obtain the RGB enhanced features; the depth features after attention enhancement are adaptively selected through the Sigmoid function, and multiplied and added with the preliminary enhanced depth features to obtain the depth enhanced features; The RGB enhanced features and depth enhanced features are concatenated and convolved to obtain the final fusion features.

[0012] Optionally, the method further includes determining a deviation between the RGB-D salient target prediction map and the RGB-D salient target annotation map based on the following formula: ; ; ; in, G is the RGB-D salient object annotation map, S i For decoder i Level RGB-D salient object prediction map, λ i The decoder i The weight of the level, is the binary cross entropy loss, is the IoU loss, is the deviation between the RGB-D salient target prediction map and the RGB-D salient target annotation map, w is the pixel width, h is the pixel height.

[0013] The RGB-D salient object detection method provided by the embodiment of the present invention has the following beneficial effects compared with the prior art: The present invention realizes the deep fusion of RGB features and depth information through the collaboration of dynamic feature calibration and edge perception constraints, balances global reasoning and local optimization, integrates color texture and spatial geometry information, and enhances global context perception; introduces the edge information of the original image, fine-tunes the decoding features, and solves the boundary ambiguity problem; this collaboration enables the cross-modal saliency detection network to accurately capture global semantics and precisely locate target boundaries in complex scenes, achieving an effective balance between global and local information, significantly improving the visual integrity and detail fidelity of saliency detection, and enhancing the accuracy of saliency detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the RGDCNet framework of an RGB-D salient object detection method provided in one embodiment; Figure 2 A schematic diagram of a CMCM framework of an RGB-D salient object detection method provided in one embodiment; Figure 3 A schematic diagram of a CMAM framework for an RGB-D salient object detection method provided in one embodiment; Figure 4 A schematic diagram of a DAFM framework of an RGB-D salient object detection method provided in one embodiment; Figure 5 FIG1 is a schematic diagram of comparative analysis of saliency detection of an RGB-D salient object detection method provided in one embodiment; Figure 6 A diagram showing the experimental design results of a backbone network for an RGB-D salient object detection method provided in one embodiment; Figure 7 This is a diagram showing the CMCM feature calibration effect of an RGB-D salient object detection method provided in one embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0016] An embodiment of the present invention provides an RGB-D salient object detection method, the method comprising: Obtain color images, depth information, and corresponding RGB-D salient object annotations from the RGB-D saliency detection dataset.

[0017] The color image and depth information are fed into a cross-modal saliency detection network to obtain an RGB-D salient object prediction map. The cross-modal saliency detection network is trained to minimize the deviation between the RGB-D salient object prediction map and the RGB-D salient object annotation map, resulting in a trained cross-modal saliency detection network.

[0018] The color image to be processed and the depth information to be processed are input into a trained cross-modal saliency detection network. The trained cross-modal saliency detection network comprises a parallel encoder, a cross-modal calibration module, a cross-modal attention enhancement module comprising a first interaction branch and a second interaction branch, a dual-attention fusion module, and a decoder, all connected in sequence. The parallel decoder comprises an RGB encoder and a depth encoder, with an edge refinement perception module added between the RGB encoder input and the decoder output. The RGB encoder extracts RGB features from the color image, while the depth encoder extracts depth features from the depth information. The cross-modal calibration module transforms and concatenates the RGB and depth features to produce an initial fused feature. The first interaction branch normalizes the weights of the initial fused RGB features to produce an RGB feature stream. The second interaction branch normalizes the weights of the initial fused features and the RGB features to produce a depth feature stream. The dual-attention fusion module concatenates the RGB and depth feature streams to produce a final fused feature. The decoder decodes the final fused feature to produce a final decoded feature. The edge refinement perception module processes the color image and the final decoded feature to produce an RGB-D salient object recognition map.

[0019] The specific implementation is as follows: 1. Cross-modal Saliency Detection Network (RGDCNet).

[0020] The RGDCNet framework proposed in this invention is as follows Figure 1 As shown in Figure 1, a dual-stream encoder-decoder architecture is used to implement RGB-D salient object detection through multi-level and multi-layer feature interaction and attention mechanism. Specifically, the network consists of the following key components: first, the network uses two independent Swinv2 Transformer encoders to process RGB and depth information respectively, extracting multi-scale features, which are denoted as and , where d belongs to {1, 2, 3, 4}. To address the issue of unstable depth image quality, a Cross-Modal Calibration Module (CMCM) was designed. By inputting RGB and depth features from each stage, CMCM transfers RGB information to depth features. It also integrates channel attention and multi-scale spatial attention mechanisms to adaptively calibrate depth features, enhance their expressiveness, and suppress irrelevant feature noise. Secondly, a Cross-Modal Attention Module (CMAM) was designed and implemented. It uses high-level decoder features and upper-level semantic features to guide encoding features, optimizing low-level encoding features and achieving both RGB-to-depth and depth-to-RGB feature enhancement. Furthermore, a Dual Attention Fusion Module (DAFM) was designed, combining channel attention and spatial attention mechanisms to adaptively adjust feature weights for fusion. The network decoder uses a multi-stage decoder with the same modular structure as the SwinTransformer, gradually restoring feature resolution from high to low layers. Finally, the Edge Refinement Awareness (ERA) module is used to input the original RGB input (color image) and the depth image, and the channel attention mechanism is used to dynamically calibrate the modal weights. Finally, the multi-scale context information is fused through residual connections and combined with the final decoded features to generate the final RGB-D salient object recognition map.

[0021] The spatial information of RGB and depth features is aligned, and then the modal weights are dynamically calibrated using the channel attention mechanism. Finally, multi-scale contextual information is fused through residual connections to improve the representation ability of cross-modal features.

[0022] 2. Cross-modal calibration module (CMCM).

[0023] How to fuse RGB and depth features while reducing the interference of low-quality depth information on prediction results is a difficult problem that needs to be solved urgently. Existing methods often directly fuse the features of the two modalities without fully considering the instability that may be caused by fluctuations in the quality of depth information. This paper proposes CMCM, which dynamically optimizes depth features using RGB features as a guide. This module can effectively compensate for information deviations in the depth modality caused by factors such as illumination changes and sensor noise, ensuring the reliability and usability of depth features in saliency detection tasks, thereby improving the robustness of the model in complex environments.

[0024] like Figure 2 As shown, CMCM first transforms the input RGB features through a linear layer and deep features Feature transformation: ; in, is the RGB feature, is the deep feature, is the RGB feature after feature transformation, is the deep feature after feature transformation.

[0025] In order to make full use of RGB features to guide the enhancement of deep features, the channel attention mechanism (CA) and spatial attention mechanism (SA) are used to further enhance the features. The channel attention mechanism focuses on the relationship between channels of the features and is calculated as follows:

[0026] ; in, σ is the activation function. The channel attention weights are then applied to the deep features to achieve channel-level feature enhancement. At the same time, the spatial attention mechanism is used to highlight the spatial information of the salient regions:

[0027] ; Through element-wise multiplication operations, the channel attention and spatial attention are multiplied with the deep features respectively: ; in, is the deep feature processed by the channel attention mechanism, is the enhanced deep feature, LinearLayer is the fully connected layer, CA is the spatial attention mechanism, and SA is the channel attention mechanism.

[0028] In order to achieve adaptive fusion, a feature self-calibration mechanism is introduced. First, the enhanced depth features and RGB features are spliced ​​together through the splicing operation C. This process can be expressed as:

[0029] ; Among them, C is the splicing operation, The result of splicing.

[0030] Subsequently, the concatenated result and the deep features after feature transformation are adaptively weighted by two convolution branches (CBR) and learnable weight α. Perform residual connection to improve the flow of information.

[0031] ; in CBR Represents a "Conv+BN+ReLU" operation, where the first branch uses 1×1 convolution and the second branch uses 3×3 convolution to capture feature information of different scales. Parameters is a learnable weight used to balance the contribution of the two branches.

[0032] Through this cross-modal calibration and feature enhancement mechanism, RGB features can be effectively used to guide the enhancement of depth features, enabling the model to have stronger salient object detection capabilities in complex RGB-D scenes, especially maintaining stable performance when the quality of depth information is poor.

[0033] 3. Cross-modal Attention Enhancement Module (CMAM).

[0034] To improve the robustness of models when processing multimodal data, existing methods require more sophisticated fusion strategies. Currently, common methods, such as simple feature concatenation or weighted fusion, fail to fully capture the dynamic complementary nature of RGB and depth modalities in different scenarios. This results in significant degradation of model performance when the quality of information from one modality degrades. To address this issue, CMAM was proposed. It establishes effective information interaction between RGB and depth features through a bidirectional attention mechanism and high-level feature guidance, thereby improving feature representation capabilities.

[0035] like Figure 3 As shown, CMAM receives multiple inputs: the RGB features of the current layer , deep features , decoder features , and high-level features and .

[0036] First, the high-level features and decoder features are upsampled and preliminarily convolved: ; in, To decode the features, For RGB features Decoded RGB high-level features, is the deep high-level feature of the initial fusion feature after preliminary decoding, U is the upsampling operation, and S is the Sigmoid activation function.

[0037] Next, we construct two interactive branches to process RGB and depth features respectively. The RGB features are copied into two parts. One part is element-wise multiplied with the decoder features, then concatenated with the high-level semantic features and convolved. Finally, a residual connection is made with the other part of the RGB features. The details are as follows:

[0038] ; Among them, × represents element-wise multiplication, C Represents a feature concatenation operation, and + represents element-wise addition.

[0039] The same operation is performed for deep features: ; Then, the enhanced features and Mapped to queries Q ,key K Sum V express: ; Through matrix multiplication and Softmax function, the attention weight is calculated and the final enhanced features are generated: ; in, To enhance the initial fusion features, To enhance RGB features, To enhance the query of RGB features, To enhance the RGB feature key, To enhance the value of RGB features, To enhance the query of the initial fusion features, To enhance the key of the initial fusion feature, To enhance the value of the initial fusion feature, To enhance the query weight of RGB features, To enhance the key weight of RGB features, To enhance the value weight of RGB features, To enhance the query weight of the initial fusion features, To enhance the key weights of the initial fusion features, To enhance the value weight of the initial fusion feature, is the RGB feature stream, is the deep feature flow.

[0040] The matrix multiplication operation captures the correlation between features, and the Softmax function is used to normalize the attention weights.

[0041] Through this two-way interaction mechanism, the CMAM module achieves efficient complementary enhancement of RGB and depth information. As a modulation signal, it guides the RGB features to focus on the areas with significant depth changes; at the same time, the high-level features and It provides semantic-level guidance to help the features at the current level learn more discriminative representations.

[0042] 4. Dual Attention Feature Fusion Module (DAFM).

[0043] In the task of RGB-D saliency detection, the proper fusion of RGB and depth features is crucial for the final inference of salient objects. Existing fusion methods mostly employ simple concatenation or weighted summation strategies, which struggle to effectively leverage the complementary information between RGB and depth modalities. Furthermore, due to the significant differences in the feature distributions of RGB and depth data, direct fusion can lead to information loss or interference. To address this issue, we propose DAFM, which achieves more robust cross-modal feature fusion through a cascaded channel-attention and spatial-attention mechanism.

[0044] like Figure 4 As shown, the DAFM module receives two input features: RGB feature stream and deep feature flow The module adopts a symmetrical dual-stream structure to enhance and fuse the two modal features respectively.

[0045] First, an average pooling operation is performed on the RGB feature stream and the depth feature stream to extract global context information: ; in, A Represents an average pooling operation.

[0046] Then, the pooled feature streams are concatenated and transformed through the convolution block to obtain the intermediate fusion features: ; in, C For splicing operations, CBR is the Conv1×1+BN+ReLU convolution block, is the RGB feature stream after pooling, is the deep feature flow after pooling.

[0047] Next, the intermediate fusion features are element-wise multiplied with the RGB feature stream and the depth feature stream respectively, and added to the original features to obtain preliminary enhanced RGB features and preliminary enhanced depth features: ; in, represents element-wise multiplication, and + represents element-wise addition.

[0048] In order to further enhance the feature representation capability, the cascaded channel attention (CA) and spatial attention (SA) mechanisms are introduced to enhance the RGB features and depth features through channel attention and spatial attention, and obtain the attention-enhanced RGB features and attention-enhanced depth features. The channel attention is shown as follows:

[0049] ; The spatial attention enhancement is as follows: ; Among them, CA represents the channel attention module and SA represents the spatial attention module.

[0050] Finally, the RGB features after attention enhancement are adaptively selected using the Sigmoid function, multiplied and added with the preliminary enhanced RGB features to obtain the RGB enhanced features. The depth features after attention enhancement are adaptively selected using the Sigmoid function, multiplied and added with the preliminary enhanced depth features to obtain the depth enhanced features. The RGB enhanced features and the depth enhanced features are concatenated and convolved to obtain the final fused features.

[0051] ; Among them, S represents the Sigmoid activation function.

[0052] The DAFM module achieves complementary enhancement of RGB and depth features through this cascaded two-stream attention mechanism. The channel attention mechanism focuses on the inter-channel relationships of features, enhancing semantically relevant channel responses; the spatial attention mechanism highlights the spatial information of salient regions, improving the model's perception of target areas. Adaptive sigmoid enhancement further enhances feature discriminability, enabling the model to more accurately locate salient objects.

[0053] 5. Loss function.

[0054] To optimize the performance of RGB-D saliency detection models, we employ a widely used loss function design that combines Binary Cross-Entropy (BCE) and Intersection-over-Union (IoU) loss to comprehensively measure the pixel-level, structural-level, and region-level differences between the predicted and true saliency maps. This loss combination has been adopted by multiple saliency detection methods and effectively improves the prediction accuracy and boundary quality of salient objects.

[0055] The binary cross entropy loss is used to calculate the pixel-level difference between the predicted saliency map S^\hat{S}S^ and the true saliency map G, which is defined as follows: ; BCE loss can effectively optimize pixel-level saliency segmentation results, but because it does not consider the global structural information of the image, it may produce large errors near the object boundary.

[0056] IoU loss (WIoU) is able to enhance supervision of difficult-to-detect areas, especially in the target boundary area. It is defined as follows:

[0057] ; This weighting strategy can enhance the focus on salient regions and boundary information, making the loss calculation more consistent with the distribution characteristics of actual salient targets. A multi-task learning strategy is adopted to comprehensively learn hierarchical saliency information from multiple layers, and the output is jointly trained with multiple auxiliary side outputs to form the model output. Therefore, the complete loss of model training can be expressed as:

[0058] ; in G To mark the results of significance, S i For decoder i The saliency prediction map of the level prediction is upsampled to form a mask with the same resolution as the ground truth. λ i The decoder i The weight of the level. Weight set λ Following {0.8, 0.6, 0.4, 0.2}, it is associated with the network hierarchy. This loss function design has been verified in multiple RGB-D saliency detection tasks and can effectively optimize the regional integrity, boundary clarity, and global structural information of salient objects, thereby improving the quality of the final prediction results.

[0059] 6. Comparative experiment.

[0060] 6.1 Dataset

[0061] Experiments are conducted on seven widely used RGB-D saliency detection datasets to verify the effectiveness of the proposed RGDCNet.

[0062] NJUD contains 2003 pairs of stereo images, released in 2014. The images, sourced from internet resources and 3D movie clips, cover a wide range of objects and complex scenes, along with their corresponding real-world depth maps. Its unique value lies in its inclusion of stereo images captured from multiple perspectives, providing rich data for multi-view saliency detection research.

[0063] SIP is a high-quality dataset containing 929 images covering a wide range of human poses and actions. The data covers complex lighting conditions, occluded objects, and diverse object types, combining realism and practicality, making it one of the most popular SOD evaluation datasets.

[0064] NLPR, released in 2014, consists of 1,000 stereo images covering 11 different indoor and outdoor scenes. Its high-quality image annotations and balanced scene distribution have made it widely used in SOD tasks.

[0065] SSD is a small-scale dataset that contains 80 natural images from left and right perspectives selected from three stereoscopic movies.

[0066] The DUT consists of 1,200 images, 800 of which are indoor scenes and 400 are outdoor scenes, with corresponding depth maps provided.

[0067] STERE, the first stereo image collection for saliency analysis, initially contained 1,000 images. After the official update, 797 images were retained, covering a variety of indoor and outdoor scenes.

[0068] LFSD contains 100 extremely challenging background and foreground images and supports multi-view saliency analysis, but the scenes are highly repetitive.

[0069] For fair comparison, following the common setup of previous studies, the training set includes 700 images from NLPR, 800 pairs from DUT, and 1,485 samples from NJUD. The remaining images (including corresponding depth maps) and the other three datasets are used for testing. Data augmentation techniques, including random horizontal flipping, random rotation (±15°), and random cropping, are also used to improve the model's generalization ability.

[0070] 6.2 Evaluation Metrics

[0071] This study uses four evaluation metrics to measure model performance, including Mean Absolute Error (MAE), Maximum F-measure, E-measure, and S-measure.

[0072] MAE is used to quantify the average absolute difference between the predicted saliency map and the ground truth map (GT), and its calculation formula is: ; Where P(x,y) and G(x,y) represent the predicted value and true value at the pixel position (x,y), respectively, and w and h are the width and height of the image.

[0073] The maximum F-measure is the harmonic mean of precision and recall, which more fairly evaluates the model's ability to capture salient regions. The model performance is evaluated by weighted precision and recall, which is expressed as:

[0074] ; The value is 0.3.

[0075] E-measure is a perceptually inspired metric used to measure the global and local similarity between the predicted image and the ground-truth image, and is defined as: ; in, Represents the matrix value at pixel (x, y), where w and h are the height and width of the image respectively.

[0076] The S-measure is used to evaluate the spatial structural similarity between the predicted graph and the true value graph, and is calculated as follows: ; in, and They represent the object-aware and region-aware structural similarity respectively, and α is usually set to 0.5.

[0077] 6.3 Experimental Setup

[0078] All experiments were conducted on an NVIDIA RTX4090 GPU. The model was implemented using the PyTorch framework. The model used the Swinv2Transformer pre-trained model to extract RGB and depth features, respectively. The training strategy employed the Adam optimizer, with an initial learning rate of 1e-4 and a weight decay of 1e-4. A cosine annealing strategy was used, with a minimum learning rate of 1e-6, a batch size of 8, and 200 training epochs. Data augmentation employed random horizontal flipping, random rotation (±10°), random scaling (0.8-1.2), and a uniform input size of 256*256.

[0079] 6.4 Comparison of experimental results.

[0080] To verify the effectiveness of the proposed model, a comprehensive comparative analysis was conducted with the current state-of-the-art models on multiple RGB-D salient object detection datasets. In the experiments, the proposed RGDCNet was compared with eight state-of-the-art RGB-D saliency detection methods, including CNN-based deep learning methods and Transformer-based RGB-D saliency detection models. These methods cover different strategies such as encoding-decoding structures, attention mechanisms, and multimodal feature fusion. These include AirSOD, MIRV, MAGNetSPCNet, FCFNet, ARDINet, EatNet, and FCDHNet. For a fair comparison, their performance was calculated based on saliency maps. All evaluation metrics were calculated by the evaluation tool. The experimental results show that the present invention outperforms the existing technology in all evaluation metrics, demonstrating the effectiveness and superiority of the present invention.

[0081] Table 1 Comparison of RGB-D salient object detection performance Table 1 presents the results of quantitative evaluation. Experiments were conducted on multiple public benchmark datasets, and performance was compared using four metrics: MAE (mean absolute error), maximum F-measure (Fβ), maximum E-measure (Em), and S-measure (Sm). The experimental results demonstrate that the proposed model achieves excellent performance across multiple datasets. In particular, in datasets containing complex backgrounds, low-quality depth maps, or multi-object scenes, the proposed model demonstrates greater robustness and generalization capabilities than other methods. For example, on the challenging SIP dataset, the proposed method achieves significant improvements across all evaluation metrics compared to the best existing methods, fully demonstrating the model's stability and effectiveness in complex environments.

[0082] In the qualitative evaluation, a comparative analysis of saliency detection was conducted for multiple typical scenarios, including challenging situations such as small objects, multiple objects, low-quality depth maps, and complex boundaries. Figure 5 Compared with other methods, the proposed model is more accurate in locating salient objects and effectively preserves details in boundary areas. Furthermore, even in the case of low-quality depth maps or those with significant noise, the model can still maintain high detection accuracy, fully demonstrating its robustness in complex scenarios.

[0083] In summary, comprehensive quantitative and qualitative analysis demonstrates the effectiveness and competitiveness of the proposed model in RGB-D saliency detection. The model maintains excellent performance in multiple complex scenarios while offering lower computational overhead and faster inference speed, demonstrating high application value.

[0084] 6.5 Ablation Experiment

[0085] To validate the effectiveness of the key components of the proposed RGDCNet model, a series of ablation experiments were conducted. These experiments aim to quantify the contribution of each module to the overall performance and demonstrate the rationality of the design. All ablation experiments were conducted on the following two datasets (i.e., DUT and STERE) to confirm the effectiveness of each component and analyze their contribution to the overall performance of the model. The following evaluation metrics were used: mean absolute error (MAE), F-measure (Fβ), structural similarity (S-measure), and E-measure. All experiments used the same training settings, including parameters such as learning rate, batch size, and optimizer, to ensure fair comparison.

[0086] 6.5.1 Validity of each module.

[0087] We first analyzed the contributions of four key components in RGDCNet: the Cross-Modal Attention Module (CMAM), which implements bidirectional attention interaction between RGB and depth features; the Cross-Modal Calibration Module (CMCM), which optimizes depth features and reduces noise and distortion; and the Dual Attention Fusion Module (DAFM), which fuses multimodal features through channel-wise and spatial attention mechanisms. Table 2 shows the ablation results for each component.

[0088] Table 2 Results of ablation experiments on key components Table 2 shows that the addition of each component improves performance. In particular, the introduction of CMAM significantly reduces MAE and improves F-measure, demonstrating that cross-modal attention interaction is crucial for effectively utilizing RGB and depth information. The addition of CMCM further improves performance, demonstrating the necessity of deep feature optimization. The introduction of DAFM and ERAA enhances feature fusion and edge detail preservation, respectively, leading to the model achieving optimal performance.

[0089] 6.5.2 Backbone network experiment design.

[0090] To explore the impact of different backbone networks on the RGB-D salient object detection task. The current model uses SwinV2 Transformer as the encoder, which is replaced with different backbone networks, mainly ResNet-50, ConvNeXt-Tiny, Swin-T (Swin Transformer Tiny), etc., to analyze the impact of different backbone networks on the final detection performance. For fair evaluation, the decoder structure, feature fusion module (CMCM, CMAM, DAFM) and training strategy are kept unchanged, only the encoder is replaced, and the experimental results are recorded. The experimental results are shown in the figure. Figure 6 shown.

[0091] We further investigated the impact of different backbone networks on RGB-D salient object detection, using ResNet-50, Convnext, ConvNeXt-Tiny, Swin-T, and SwinV2 for experimental analysis. The results showed that Transformer-based backbone networks (Swin-T and SwinV2) significantly outperformed CNN architectures (ResNet and EfficientNet), demonstrating the Transformer's superiority over CNNs in modeling global features. SwinV2 achieved the best performance across all evaluation metrics, owing to its enhanced ability to learn multi-scale features. In contrast, ResNet-50 achieved the lowest performance, primarily due to its shallower network structure, which inadequately models the characteristics of complex salient objects. ConvNeXt, an improved CNN architecture, narrowed the performance gap with the Transformer to some extent, but still lagged behind SwinV2. Overall, the experimental results validate the effectiveness of the Swin Transformer as an encoder for the RGB-D SOD task.

[0092] 6.5.3 Loss Function Experiment

[0093] To evaluate the impact of different loss function configurations on the RGB-D salient object detection task, we designed a series of ablation experiments, examining the use of only the binary cross entropy (BCE) loss, only the IoU loss, both BCE and IoU losses without multi-scale supervision, and a combination of BCE and IoU losses with multi-scale supervision. The experiments were conducted on the DUT and STERE datasets. All experiments were conducted under the same training environment to ensure fairness and comprehensively measure the quality of salient object detection.

[0094] Table 3 Effect of different loss function combinations on model performance Table 3 shows the impact of different loss function combinations on model performance. It can be observed that the model using the BCE loss alone performs relatively poorly. This is primarily because the BCE loss only optimizes pixel-level classification and fails to adequately address the global structure of the target region, resulting in a high false detection rate. In contrast, the model using the IoU loss alone performs better in terms of regional consistency, with improvements in both the F-measure and E-measure, but still has some deficiencies in terms of boundary detail. When the BCE and IoU losses are combined, model performance is further improved, with both boundary preservation and salient region integrity improved. However, there is still a gap compared to the final method. After introducing multi-scale supervision, the model achieves optimal performance across all evaluation metrics, with particularly significant results in optimizing salient object boundaries.

[0095] 6.5.4 CMCM Characteristic Calibration Effect.

[0096] The effectiveness of the proposed CMCM module in enhancing the performance of saliency detection is demonstrated by calibrating RGB features with deep features. To verify its impact, qualitative results of the feature maps before and after calibration are shown, as shown in Figure 2. Figure 7 As shown in Figure 3. Specifically, RGB features are initially processed and fused with deep features, and the CMCM module refines the combined features to align them more effectively. The results clearly show that the calibrated feature maps exhibit more accurate and coherent spatial representations, improve the accuracy of saliency detection, and reduce the inconsistency between RGB and depth modalities. These observations highlight the key role of the CMCM module in improving feature alignment and overall model performance.

[0097] The above ablation experiments verify the effectiveness and necessity of each key component in RGDCNet. The results show that cross-modal feature calibration enhancement, attention interaction, multimodal feature fusion, and edge detail preservation all contribute significantly to the model performance. The superiority of the two-stream symmetric encoder design and bidirectional cross-attention mechanism is also verified. These findings provide valuable design guidance for RGB-D saliency detection.

[0098] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. A RGB-D salient object detection method, characterized in that: include: Obtain color images, depth information, and corresponding RGB-D salient object annotations from the RGB-D saliency detection dataset; The color image and depth information are input into the cross-modal saliency detection network to obtain an RGB-D salient object prediction map; the cross-modal saliency detection network is trained with the goal of minimizing the deviation between the RGB-D salient object prediction map and the RGB-D salient object annotation map to obtain a trained cross-modal saliency detection network; The color image to be processed and the depth information to be processed are input into a trained cross-modal saliency detection network, wherein the trained cross-modal saliency detection network includes: a parallel encoder, a cross-modal calibration module, a cross-modal attention enhancement module including a first interactive branch and a second interactive branch, a dual attention fusion module, and a decoder connected in sequence; the parallel decoder includes an RGB encoder and a depth encoder, and an edge refinement perception module is added between the input end of the RGB encoder and the output end of the decoder; the RGB features of the color image are extracted by the RGB encoder, and the depth features of the depth information are extracted by the depth encoder; the RGB features and the depth features are transformed and spliced ​​by the cross-modal calibration module to obtain initial fused features; the initial fused RGB features are weight-normalized by the first interactive branch to obtain an RGB feature stream; the initial fused features and RGB features are weight-normalized by the second interactive branch to obtain a depth feature stream; the RGB feature stream and the depth feature stream are spliced ​​by the dual attention fusion module to obtain a final fused feature; the final fused feature is decoded by the decoder to obtain a final decoded feature; the color image and the final decoded feature are processed by the edge refinement perception module to obtain an RGB-D salient object recognition map.

2. The RGB-D salient object detection method according to claim 1, wherein: The cross-modal calibration module performs feature transformation and concatenation on the RGB features and the depth features to obtain the initial fusion features, specifically including: The RGB features and depth features are transformed through the fully connected layer based on the following formula: ； in, is the RGB feature, is the deep feature, is the RGB feature after feature transformation, is the depth feature after feature transformation; The deep features after feature transformation are enhanced through the channel attention mechanism and the spatial attention mechanism based on the following formula: ； in, is the deep feature processed by the channel attention mechanism, is the enhanced deep feature, LinearLayer is the fully connected layer, CA is the spatial attention mechanism, and SA is the channel attention mechanism; The enhanced depth features and RGB features are spliced ​​together through the following splicing operation to obtain the splicing result: ； Among them, C is the splicing operation, is the splicing result; The concatenation result and the deep features are residually connected based on the following formula to obtain the initial fusion features: ； in, CBR Represents Conv+BN+ReLU operation, are learnable weights.

3. The RGB-D salient object detection method according to claim 2, wherein: The initial fusion features are weighted normalized by the first interaction branch to obtain an RGB feature stream; The initial fusion features and RGB features are weighted normalized through the second interaction branch to obtain a deep feature stream, which specifically includes: The initial fusion features and RGB features are upsampled and preliminarily convolved based on the following formula: ； in, To decode the features, It is the RGB high-level feature after the initial decoding of RGB features. is the deep high-level feature of the initial fusion feature after preliminary decoding, U is the upsampling operation, S is the Sigmoid activation function; The initial fusion feature is processed by the first interactive branch to obtain an enhanced initial fusion feature; the RGB feature is processed by the second interactive branch to obtain an enhanced RGB feature; The enhanced initial fusion features and enhanced RGB features are mapped to query, key and value respectively. The attention weights are determined and the RGB feature stream and the depth feature stream are generated through matrix multiplication and Softmax function based on the following formula: ； in, is the RGB feature stream, is the deep feature flow.

4. The RGB-D salient object detection method according to claim 3, wherein: The enhanced initial fusion features and enhanced RGB features are mapped to query, key and value respectively based on the following formula: ； in, To enhance the initial fusion features, To enhance RGB features, To enhance the query of RGB features, To enhance the RGB feature key, To enhance the value of RGB features, To enhance the query of the initial fusion features, To enhance the key of the initial fusion feature, To enhance the value of the initial fusion feature, To enhance the query weight of RGB features, To enhance the key weight of RGB features, To enhance the value weight of RGB features, To enhance the query weight of the initial fusion features, To enhance the key weights of the initial fusion features, is the value weight to enhance the initial fusion feature.

5. The RGB-D salient object detection method according to claim 1, wherein: The dual attention fusion module is used to splice the RGB feature stream and the depth feature stream to obtain the final fusion feature, which specifically includes: Perform average pooling on the RGB feature stream and the depth feature stream to extract global context information; The pooled feature streams are spliced ​​and transformed through convolution blocks to obtain intermediate fusion features; Perform element-wise multiplication of the intermediate fusion features with the RGB feature stream and the depth feature stream respectively, and add them to the original features to obtain preliminary enhanced RGB features and preliminary enhanced depth features; Enhance the RGB features and depth features through channel attention and spatial attention to obtain attention-enhanced RGB features and attention-enhanced depth features; The RGB features after attention enhancement are adaptively selected through the Sigmoid function, and multiplied and added with the preliminary enhanced RGB features to obtain the RGB enhanced features; the depth features after attention enhancement are adaptively selected through the Sigmoid function, and multiplied and added with the preliminary enhanced depth features to obtain the depth enhanced features; The RGB enhanced features and depth enhanced features are concatenated and convolved to obtain the final fusion features.

6. The RGB-D salient object detection method according to claim 1, wherein: It also includes determining the deviation between the RGB-D salient object prediction map and the RGB-D salient object annotation map based on the following formula: ； ； ； in, G is the RGB-D salient object annotation map, S i For decoder i Level RGB-D salient object prediction map, λ i The decoder i The weight of the level, is the binary cross entropy loss, is the IoU loss, is the deviation between the RGB-D salient target prediction map and the RGB-D salient target annotation map, w is the pixel width, h is the pixel height.

Images