RGB-D salient object detection method and system based on cross-modal association learning

Abstract

The present invention proposes an RGB-D salient target detection method and system based on cross-modal association learning. The method comprises: constructing a feature extraction network of RGB modality and deep modality based on a dual-stream encoder architecture; using an adaptive cross-modal fusion module to perform feature fusion to obtain cross-modal fusion features; using element-by-element multiplication to perform attention enhancement on refined large-scale features and small-scale up-sampled features to obtain output fusion features of a multi-scale feature aggregation module; using an upsampling layer and a convolutional layer to predict the object boundary of the output fusion features of the multi-scale feature aggregation module to obtain the features of an edge prediction map; performing point-by-point addition operations on salient features to obtain a final salient prediction map; inputting RGB-D data into a network model for training to generate RGB-D salient target detection results; the present invention improves the accuracy and robustness of salient target detection by mining the correlation and complementarity between modalities and fully integrating the characteristic information of different modalities.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method and system for detecting salient RGB-D objects using cross-modal association learning. Background Art

[0002] In the field of computer vision, RGB-D salient object detection has gradually become an important research direction in recent years. Salient object detection aims to accurately locate and segment attractive or important areas in the scene by analyzing image or video data, and provide support for subsequent high-level visual tasks (such as object recognition, scene understanding, object segmentation, etc.). Compared with traditional single-modal salient object detection, RGB-D salient object detection can simultaneously utilize the information of RGB images and depth images. RGB images provide rich color and texture features, while depth images provide geometric and distance information of objects in the scene. This multimodal characteristic makes RGB-D salient object detection have greater potential and application value in complex scenes, such as autonomous driving, robot navigation, medical image analysis, and virtual reality.

[0003] Traditional salient object detection algorithms usually rely on hand-crafted features or simple fusion strategies, but they have limited use of inter-modal correlation information and are difficult to achieve robust performance in diverse scenarios. In recent years, the rapid development of deep learning technology has promoted the progress of salient object detection, especially the powerful capabilities of convolutional neural networks (CNNs) in feature extraction and modeling, which has enabled deep learning-based RGB-D salient object detection to achieve remarkable results. However, CNNs mainly focus on local feature relationships and lack the ability to model global dependencies between modalities and cross-modal characteristics, which to some extent limits the detection accuracy and generalization ability. The Transformer architecture was introduced into computer vision tasks due to its success in the field of natural language processing. Through its powerful attention mechanism, it can effectively model global dependencies within and between modalities. This advantage has led to its gradual application in RGB-D salient object detection tasks. However, the high computational complexity and high memory requirements of Transformer have largely limited its practicality in real-time detection tasks. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose an RGB-D salient object detection method and system based on cross-modal association learning to solve the above technical problems.

[0005] The present invention proposes a cross-modal association learning RGB-D salient object detection method, the method comprising the following steps:

[0006] Step 1: construct a feature extraction backbone network based on the dual-stream encoder module and the adaptive cross-modal fusion module; construct a salient object decoder based on the hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute the RGB-D salient object detection network model;

[0007] Step 2: Using the information encoding module in the dual-stream encoder module to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and using the adaptive cross-modal fusion module to perform hierarchical feature fusion on the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image to obtain fused cross-modal fusion features;

[0008] Step 3: Represent the adjacent features of the fused cross-modal fusion features to obtain adjacent scale features, and fuse the adjacent scale features using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features;

[0009] Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features;

[0010] The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map;

[0011] Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features;

[0012] The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain output fusion features of the hierarchical multi-scale feature aggregation module;

[0013] Step 4: Use the upsampling layer and the convolution layer to predict the object boundary based on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the features of the edge prediction map;

[0014] Step 5: Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map;

[0015] Step 6: Input the RGB-D data into the RGB-D salient object detection network model for training, and combine the features of the edge prediction map and the final saliency prediction map to generate the RGB-D salient object detection results.

[0016] The present invention also proposes a cross-modal association learning RGB-D salient object detection system, the system comprising:

[0017] Network building blocks for:

[0018] A feature extraction backbone network is constructed based on a dual-stream encoder module and an adaptive cross-modal fusion module. A salient object decoder is constructed based on a hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute an RGB-D salient object detection network model.

[0019] Feature extraction module for:

[0020] The information encoding module in the dual-stream encoder module is used to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are hierarchically fused using an adaptive cross-modal fusion module to obtain fused cross-modal fusion features;

[0021] Saliency calculation module, used to:

[0022] The adjacent features of the fused cross-modal fusion features are represented to obtain adjacent scale features, and the adjacent scale features are fused using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features;

[0023] Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features;

[0024] The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map;

[0025] Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features;

[0026] The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain output fusion features of the hierarchical multi-scale feature aggregation module;

[0027] Edge-aware optimization module for:

[0028] The output fusion features of the hierarchical multi-scale feature aggregation module are used to predict the object boundary using the upsampling layer and the convolution layer to obtain the features of the edge prediction map.

[0029] Offline training module for:

[0030] Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map;

[0031] Salient object detection module, used to:

[0032] The RGB-D data is input into the RGB-D salient object detection network model for training, and the features of the edge prediction map and the final saliency prediction map are combined to generate the RGB-D salient object detection results.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention uses an adaptive cross-modal fusion module to deeply explore the correlation and complementarity between RGB modality and depth modality, effectively improving the expression ability of multimodal features, thereby achieving more accurate salient object detection;

[0035] 2. The present invention uses edge-guided branches to extract edge features of salient targets and combines them with structural quality perception optimization objectives, significantly alleviating the edge structure distortion problem in traditional methods, thereby maintaining the accuracy and edge integrity of detection results in complex scenarios;

[0036] 3. Based on the hierarchical multi-scale feature aggregation module, the present invention realizes the efficient integration of features of different scales and fully captures the information of salient targets at multiple levels, thereby improving the robustness and adaptability of the model in scenarios of different sizes and complexities.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of the steps of the RGB-D salient object detection method based on cross-modal association learning proposed in the present invention.

[0039] Figure 2 This is a structural diagram of the RGB-D salient object detection method based on cross-modal association learning proposed in this invention.

[0040] Figure 3 Schematic diagram of the gated bottleneck component of the RGB-D salient object detection method for cross-modal association learning proposed in this invention.

[0041] Figure 4 Schematic diagram of the principle of the modality complementary feature preservation module of the RGB-D salient object detection method with cross-modal association learning proposed in the present invention.

[0042] Figure 5 Schematic diagram of the principle of the residual feature fusion module of the RGB-D salient object detection method with cross-modal association learning proposed in the present invention.

[0043] Figure 6 Schematic diagram of the principle of the multi-scale feature aggregation module of the RGB-D salient object detection method with cross-modal association learning proposed in the present invention.

[0044] Figure 7 This is a structural diagram of the RGB-D salient object detection system with cross-modal association learning proposed in this invention. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0046] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0047] See also Figure 1 The embodiment of the present invention proposes a cross-modal association learning RGB-D salient object detection method, which includes the following steps:

[0048] Step 1: Construct a feature extraction backbone network based on the dual-stream encoder module and the adaptive cross-modal fusion module; construct a salient object decoder based on the hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute the RGB-D salient object detection network model.

[0049] In step 1, a feature extraction backbone network is constructed based on a dual-stream encoder module and an adaptive cross-modal fusion module; a salient object decoder is constructed based on a hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute an RGB-D salient object detection network model, which specifically includes the following sub-steps:

[0050] The two-stream backbone network based on SwinTransformer performs feature extraction on RGB images and depth images to obtain preliminary features of RGB images and preliminary features of depth images;

[0051] The adaptive modal relationship learning module is used to fuse the preliminary features of the RGB image and the preliminary features of the depth image to obtain the feature fusion result;

[0052] By constructing a saliency detection module, the salient target area is extracted from the feature fusion result to obtain the salient target area extraction result.

[0053] Step 2: Use the information encoding module in the dual-stream encoder module to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and use the adaptive cross-modal fusion module to perform hierarchical feature fusion on the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image to obtain a fused cross-modal fusion feature.

[0054] See also Figures 2 to 5 In step 2, the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are extracted by using the information encoding module in the dual-stream encoder module, and the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are hierarchically fused by using the adaptive cross-modal fusion module to obtain fused cross-modal fusion features, which specifically includes the following steps:

[0055] The scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are extracted by using the information encoding module to obtain the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image;

[0056] Inputting the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image into the gated bottleneck component module for mechanism processing to obtain a feature attention weight map of the RGB image and a feature attention weight map of the depth image;

[0057] The feature attention weight map of the RGB image is fused with the representation features of the multi-stage RGB image to obtain an enhanced feature expression of the RGB image; the feature attention weight map of the depth image is fused with the representation features of the multi-stage depth image to obtain an enhanced feature expression of the depth image;

[0058] The representation features of the multi-stage RGB image are convolved to obtain the query vector of the RGB mode; the feature expression of the enhanced depth image is subjected to a global average pooling operation to obtain the key vector of the depth mode;

[0059] A spatial Transformer module is constructed based on the hierarchical multi-scale feature aggregation module. The query vector of the RGB modality and the key vector of the depth modality are calculated to obtain the spatial irrelevance mapping weight matrix between the RGB modality features and the depth modality features. The spatial irrelevance mapping weight matrix is ​​then used to perform element-by-element calculations to obtain the refined RGB features in the spatial Transformer module.

[0060] A channel-independent learning module is constructed based on a hierarchical multi-scale feature aggregation module, and the query vector of the RGB modality in the channel-independent learning module is represented by using the RGB image representation feature to obtain the query vector of the RGB modality represented in the channel-independent learning module; the key vector of the deep modality in the channel-independent learning module is represented by using the feature expression of the enhanced deep image to obtain the key vector of the deep modality represented in the channel-independent learning module;

[0061] The query vector of the RGB modality represented in the channel-independent learning module and the key vector of the depth modality represented in the channel-independent learning module are sequentially reshaped and reversed to obtain the irrelevant weight matrix in the channel-independent learning module and the refined RGB features of the channel-independent learning module;

[0062] The refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module are combined to obtain refined RGB features; the representation features of the multi-stage depth image and the feature expression of the enhanced RGB image are fused using a hierarchical multi-scale feature aggregation module to obtain refined depth features;

[0063] The refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module are combined to obtain the spliced ​​refined RGB features; the spliced ​​refined RGB features are fused using the modal complementary feature retention module in the adaptive cross-modal fusion module to obtain the refined deep features; the refined deep features and the spliced ​​refined RGB features are sequentially spliced ​​and residual jumped using the residual feature fusion module in the adaptive cross-modal fusion module to obtain the fused cross-modal fusion features.

[0064] In the step of extracting the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image by using the information encoding module to obtain the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image, the corresponding relationship is as follows:

[0065] ;

[0066] in, Represents the representation features of multi-stage RGB images, Representation features of multi-stage depth images, The encoder backbone network represents the RGB image stream, The encoder backbone network represents the deep image stream, Indicates The channel index number of the stage feature map, Indicates The height of the stage feature map, Indicates The width of the stage feature map, represents an RGB image, represents the depth image, The dimension is The space of real numbers.

[0067] In the step of inputting the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image into the gated bottleneck component module for mechanism processing to obtain the feature attention weight map of the RGB image and the feature attention weight map of the depth image, the relationship between the corresponding processes is:

[0068] ;

[0069] in, Represents the feature attention weight map of the RGB image, Represents the feature attention weight map of the depth image, represents the normalization operation, represents the activation function, Indicates a convolution operation with a convolution kernel of 3x3. Representation and Different convolution kernels are 3x3 convolution operations.

[0070] In the steps of fusing the feature attention weight map of the RGB image with the representation features of the multi-stage RGB image to obtain the enhanced feature expression of the RGB image; and fusing the feature attention weight map of the depth image with the representation features of the multi-stage depth image to obtain the enhanced feature expression of the depth image, the relationship between the corresponding processes is:

[0071] ;

[0072] in, represents the feature expression of the enhanced RGB image, represents the feature expression of the enhanced depth image, Represents an element-wise multiplication operation.

[0073] In the steps of performing a convolution operation on the representation features of the multi-stage RGB image to obtain a query vector of the RGB mode; and performing a global average pooling operation on the feature expression of the enhanced depth image to obtain a key vector of the depth mode, the corresponding process has the following relationship:

[0074] ;

[0075] in, represents the query vector in RGB modality, The key vector representing the depth modality, Indicates a convolution operation with a convolution kernel of 1x1. Representation and Different convolution kernels are 1x1 convolution operations. represents the global average pooling operation, The dimension is The space of real numbers, represents a convolutional layer with a kernel of 1x1 used to process the query vector, Represents a convolutional layer with a kernel of 1x1 used to process the key vector.

[0076] In the step of building a spatial Transformer module based on the hierarchical multi-scale feature aggregation module, the query vector of the RGB modality and the key vector of the depth modality are calculated to obtain the spatial irrelevance mapping weight matrix between the RGB modality features and the depth modality features. The relationship between the corresponding process is:

[0077] ;

[0078] in, represents the spatially uncorrelated mapping weight matrix, represents a matrix tensor whose elements are all 1, represents the matrix multiplication operation, The dimension is The space of real numbers.

[0079] It should be noted that the hierarchical multi-scale feature aggregation module adopts a spatial and channel-independent learning module to capture the global interaction between different modal features. This module is a Transformer-based structure used to capture the complementary relationship between cross-modal features.

[0080] In the step of using the spatially uncorrelated mapping weight matrix to perform element-by-element calculations to obtain the refined RGB features in the spatial Transformer module, the corresponding process has the following relationship:

[0081] ;

[0082] in, Represents the refined RGB features in the spatial Transformer module.

[0083] In the step of constructing a channel-independent learning module based on a hierarchical multi-scale feature aggregation module, the query vector of the RGB modality in the channel-independent learning module is represented by using the RGB image representation feature to obtain the query vector of the RGB modality represented in the channel-independent learning module; and the key vector of the deep modality in the channel-independent learning module is represented by using the enhanced feature expression of the deep image to obtain the key vector of the deep modality represented in the channel-independent learning module. The relationship between the corresponding processes is:

[0084] ;

[0085] in, The query vector representing the RGB modality of the channel-independent learning module, The key vector representing the deep modality of the channel-independent learning module, represents the global average pooling operation, The dimension is The space of real numbers.

[0086] In the steps of successively reshaping and reversely operating the query vector of the RGB modality represented in the channel-independent learning module and the key vector of the depth modality represented in the channel-independent learning module to obtain the irrelevant weight matrix in the channel-independent learning module and the refined RGB features of the channel-independent learning module, the corresponding process has the following relationship:

[0087] ;

[0088] in, represents the uncorrelated weight matrix between the query vector of the RGB modality and the key vector of the depth modality in the channel-independent learning module, represents a matrix tensor whose elements are all 1, represents a channel-by-channel multiplication operation, represents the refined RGB features in the channel-independent learning module, The dimension is The space of real numbers.

[0089] In the step of combining the refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module to obtain the concatenated refined RGB features, the relationship between the corresponding processes is:

[0090] ;

[0091] in, represents the refined RGB features, Represents a concatenation operation.

[0092] In the step of fusing the spliced ​​refined RGB features using the modal complementary feature retention module in the adaptive cross-modal fusion module to obtain the refined depth features, and sequentially splicing and performing residual jump operations on the refined depth features and the spliced ​​refined RGB features using the residual feature fusion module in the adaptive cross-modal fusion module to obtain the fused cross-modal fusion features, the corresponding process has the following relationship:

[0093] ;

[0094] in, represents the concatenated cross-modal fusion features, Represents the residual feature fusion module in The output characteristics of the stage, represents the element-by-element addition operation, represents the batch normalization operation, represents the residual processing module used to enrich feature information, represents the refined RGB features, Represents refined deep features.

[0095] It should be noted that after obtaining the refined RGB features and depth features, the residual feature fusion module integrates the improved RGB and depth features to enable the model to learn a richer representation containing complementary information of RGB and depth modalities, such as Figure 5 By embedding cross-modal features in a unified channel space with a residual structure, the model can effectively capture cross-modal correlations and enhance the ability and robustness of salient object detection in RGB-D images.

[0096] Furthermore, The module can be represented as:

[0097] ;

[0098] in, express The image features of the module input, Represents the ReLU activation function.

[0099] Further, in Figure 2 In the figure, GBC represents the gated bottleneck module, MCFP represents the modal complementary feature preservation module, RFFM represents the residual feature fusion module, ACMF represents the adaptive cross-modal fusion module; MSFA represents the multi-scale feature aggregation module, MSFA1_1 represents the first MSFA module at the first level; UP2x represents the 2x upsampling module, CBR represents the convolutional layer, batch normalization layer and ReLU activation function mechanism processing, and Conv represents the convolutional layer.

[0100] Step 3: Represent the adjacent features of the fused cross-modal fusion features to obtain adjacent scale features, and fuse the adjacent scale features using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features;

[0101] Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features;

[0102] The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map;

[0103] Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features;

[0104] The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain the output fusion features of the hierarchical multi-scale feature aggregation module.

[0105] See also Figure 5 and Figure 6 In step 3, in the step of performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features, the relationship between the corresponding process is:

[0106] ;

[0107] in, represents the upsampled features of refined small-scale features, represents the Gelu activation function, represents a 1x1 convolutional layer, represents a bilinear upsampling operation with a scale factor of 2, Represents small-scale features; Indicates that after bilinear upsampling operation After processing, the features Indicates The index of the hierarchical multi-scale feature aggregation module, The index number of the fusion level.

[0108] It should be noted that scale feature aggregation is crucial in detecting small objects because it provides highly discriminative and robust feature representation. Multi-scale fusion should not only explore the fusion relationship of adjacent scale features, but also enable the model to learn the adaptive fusion correlation of non-adjacent scale features to enhance the performance and robustness of the detection model.

[0109] In the steps of fusing the refined large-scale features with the refined small-scale features to obtain the fused features of the refined large-scale features and the refined small-scale features, performing convolution layer calculation on the fused features of the refined large-scale features and the refined small-scale features to obtain the convolution layer calculation results of the fused features of the refined large-scale features and the refined small-scale features, and processing the convolution layer calculation results with the activation function to obtain the attention map, the corresponding process has the following relationship:

[0110] ;

[0111] in, represents the attention map, Represents large-scale features;

[0112] In the step of performing attention enhancement processing on the upsampled features of the refined large-scale features and the refined small-scale features by element-by-element multiplication to obtain enhanced large-scale features and enhanced small-scale features, the relationship between the corresponding processes is:

[0113] ;

[0114] in, represents enhanced small-scale features, represents the enhanced large-scale features, represents element-wise addition operation, Represents element-by-element multiplication operation;

[0115] In the step of fusing the enhanced large-scale features and the enhanced small-scale features using the hierarchical multi-scale feature aggregation module to obtain the output fusion features of the hierarchical multi-scale feature aggregation module, the relationship between the corresponding process is:

[0116] ;

[0117] in, Indicates The hierarchical multi-scale feature aggregation module is The output fusion features of the layer, Represents a splicing operation, Represents a 3x3 convolutional layer.

[0118] It should be noted that by using a fully connected structure with multiple hierarchical multi-scale feature aggregation modules in the decoder, the model is able to learn the correlation between multi-scale features. The output features of the last fusion level are used in the subsequent prediction branch.

[0119] Step 4: Use the upsampling layer and the convolution layer to predict the object boundary based on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the features of the edge prediction map.

[0120] See also Figure 2 In step 4, the output fusion features of the hierarchical multi-scale feature aggregation module are used to predict the object boundary using the upsampling layer and the convolution layer to obtain the features of the edge prediction map. The corresponding process has the following relationship:

[0121] ;

[0122] in, Represents edge features of half the size of the input image, represents the features of the edge prediction map of the same size as the input image, It means a convolution layer containing a batch normalization layer and a ReLU activation function 3×3 convolution kernel. Represents the batch normalization layer and ReLU activation function in the convolutional layer.

[0123] It should be noted that the image edge guidance branch is used to encourage the model to focus on learning the features of the image boundary area and enhance the model's ability to capture boundary-related information, which is crucial for accurate salient object detection. The output features of the fusion layer of the last hierarchical multi-scale feature aggregation module are used for the edge prediction branch. Since the shape of the fused feature map is 4 times smaller than that of the input image, the feature map needs to be scaled to match the size of the input image.

[0124] Step 5: Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map.

[0125] In step 5, the output fusion feature of the hierarchical multi-scale feature aggregation module is subjected to convolution layer calculation to obtain the saliency feature of the output fusion feature. The relationship between the corresponding process is:

[0126] ;

[0127] in, Indicates significant features;

[0128] In the step of performing point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map, the relationship between the corresponding process is:

[0129] ;

[0130] in, Represents the final saliency prediction map.

[0131] It should be noted that this branch uses edge features to guide the saliency prediction process, which helps the model focus on object boundaries and enhances its ability to accurately identify salient objects in complex scenes.

[0132] Step 6: Input the RGB-D data into the RGB-D salient object detection network model for training, and combine the features of the edge prediction map and the final saliency prediction map to generate the RGB-D salient object detection results.

[0133] In step 6, the RGB-D data is input into the RGB-D salient object detection network model for training, and the features of the edge prediction map and the final saliency prediction map are combined to generate the RGB-D salient object detection result, which specifically includes the following steps:

[0134] The dataset of RGB images, the dataset of saliency reference truth maps corresponding to RGB images, the dataset of depth images, and the dataset of saliency reference truth maps corresponding to depth images are made public, and the saliency reference truth maps corresponding to the RGB images in the dataset are used to perform element-by-element calculations to obtain salient target areas and non-salient areas of the RGB images; the saliency reference truth maps corresponding to the depth images in the dataset are used to perform element-by-element calculations to obtain salient target areas and non-salient areas of the depth images;

[0135] Combining the salient target area and the non-salient area of ​​the RGB image and the final saliency prediction map to obtain a reconstructed RGB image; combining the salient target area and the non-salient area of ​​the depth image and the final saliency prediction map to obtain a reconstructed depth image;

[0136] The structural similarity is used to measure the RGB image and the depth image to obtain the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image;

[0137] The structural quality perception loss of the RGB image and the structural quality perception loss of the depth image are added to obtain the overall structural quality perception loss, and the saliency reference truth map is convolved with the Laplacian kernel to obtain the edge reference truth map;

[0138] The edge reference true value map is combined with the features of the edge prediction map to calculate the edge prediction loss;

[0139] The edge prediction loss and the overall structural quality perception loss are calculated to obtain the total loss for model optimization, and the total loss is used to optimize the model to generate RGB-D salient object detection results.

[0140] In the step of using the saliency reference truth map corresponding to the RGB image in the data set to perform element-by-element calculation to obtain the salient target area and non-salient area of ​​the RGB image, the corresponding process has the following relationship:

[0141] ;

[0142] in, Indicates the salient target area, represents an RGB image, represents element-wise multiplication, represents the saliency reference truth map, represents the non-significant area, A binary inversion map representing the saliency reference truth map;

[0143] It should be noted that compared with the reference truth map, the final saliency map will inevitably be distorted, especially at the edges of objects, resulting in distortion of the edges of objects in the reconstructed RGB image. This distortion can be measured using structural similarity in image quality assessment.

[0144] In the step of measuring the RGB image and the depth image by using structural similarity to obtain the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image, the relationship between the corresponding processes is:

[0145] ;

[0146] in, represents the structural quality perceptual loss of RGB images, represents the structural quality-aware loss of the depth image, represents the reconstructed image of the RGB image, represents the structural similarity index measurement loss, A constant representing the importance of balancing losses, A constant representing the importance of balancing losses, represents the depth image, a reconstructed image representing the depth image;

[0147] In the step of adding the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image to obtain the overall structural quality perception loss, the relationship between the corresponding process is:

[0148] ;

[0149] in represents the perceived loss of overall structural quality;

[0150] In the step of performing a convolution operation on the saliency reference truth map using the Laplacian kernel to obtain the edge reference truth map, the corresponding process has the following relationship:

[0151] ;

[0152] in, represents the edge reference truth map, represents the convolution operation, represents the Laplace kernel;

[0153] In the step of calculating the edge prediction loss and the overall structural quality perception loss to obtain the total loss of model optimization, the relationship between the corresponding process is:

[0154] ;

[0155] in, represents the total loss of model optimization, represents a constant used to balance the significance loss and the marginal loss, represents a constant used to balance the significance loss and the marginal loss, represents the edge prediction loss, Represents ordinary mathematical multiplication between two numbers.

[0156] Furthermore, for the saliency prediction branch, the model proposes a structural quality-aware loss that encourages the model to learn the representation of distorted regions by measuring the structural distortion of the reconstructed image. This enables the model to better understand the contextual information of the RGB-D image, thereby improving the accuracy and stability of the salient object detection model.

[0157] See also Figure 7The present invention also provides a cross-modal association learning RGB-D salient object detection system, the system comprising:

[0158] Network building blocks for:

[0159] A feature extraction backbone network is constructed based on a dual-stream encoder module and an adaptive cross-modal fusion module. A salient object decoder is constructed based on a hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute an RGB-D salient object detection network model.

[0160] Feature extraction module for:

[0161] The information encoding module in the dual-stream encoder module is used to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are hierarchically fused using an adaptive cross-modal fusion module to obtain fused cross-modal fusion features;

[0162] Saliency calculation module, used to:

[0163] The adjacent features of the fused cross-modal fusion features are represented to obtain adjacent scale features, and the adjacent scale features are fused using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features;

[0164] Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features;

[0165] The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map;

[0166] Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features;

[0167] The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain output fusion features of the hierarchical multi-scale feature aggregation module;

[0168] Edge-aware optimization module for:

[0169] The output fusion features of the hierarchical multi-scale feature aggregation module are used to predict the object boundary using the upsampling layer and the convolution layer to obtain the features of the edge prediction map.

[0170] Offline training module for:

[0171] Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map;

[0172] Salient object detection module, used to:

[0173] The RGB-D data is input into the RGB-D salient object detection network model for training, and the features of the edge prediction map and the final saliency prediction map are combined to generate the RGB-D salient object detection results.

[0174] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0175] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0176] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A cross-modal association learning RGB-D salient object detection method, characterized in that: The method comprises the following steps: Step 1: construct a feature extraction backbone network based on the dual-stream encoder module and the adaptive cross-modal fusion module; construct a salient object decoder based on the hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute the RGB-D salient object detection network model; Step 2: Using the information encoding module in the dual-stream encoder module to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and using the adaptive cross-modal fusion module to perform hierarchical feature fusion on the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image to obtain fused cross-modal fusion features; Step 3: Represent the adjacent features of the fused cross-modal fusion features to obtain adjacent scale features, and fuse the adjacent scale features using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features; Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features; The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map; Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features; The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain output fusion features of the hierarchical multi-scale feature aggregation module; Step 4: Use the upsampling layer and the convolution layer to predict the object boundary based on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the features of the edge prediction map; Step 5: Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map; Step 6: Input the RGB-D data into the RGB-D salient object detection network model for training, and combine the features of the edge prediction map and the final saliency prediction map to generate the RGB-D salient object detection results.

2. The RGB-D salient object detection method based on cross-modal association learning according to claim 1, characterized in that: In the step 2, the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are extracted by using the information encoding module in the dual-stream encoder module, and the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are hierarchically fused by using the adaptive cross-modal fusion module to obtain fused cross-modal fusion features, which specifically includes the following steps: The scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are extracted by using the information encoding module to obtain the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image; Inputting the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image into the gated bottleneck component module for mechanism processing to obtain a feature attention weight map of the RGB image and a feature attention weight map of the depth image; The feature attention weight map of the RGB image is fused with the representation features of the multi-stage RGB image to obtain the enhanced feature expression of the RGB image; The feature attention weight map of the depth image is fused with the representation features of the multi-stage depth image to obtain an enhanced feature expression of the depth image; The representation features of the multi-stage RGB image are convolved to obtain the query vector of the RGB mode; the feature expression of the enhanced depth image is subjected to a global average pooling operation to obtain the key vector of the depth mode; A spatial Transformer module is constructed based on the hierarchical multi-scale feature aggregation module. The query vector of the RGB modality and the key vector of the depth modality are calculated to obtain the spatial irrelevance mapping weight matrix between the RGB modality features and the depth modality features. The spatial irrelevance mapping weight matrix is ​​then used to perform element-by-element calculations to obtain the refined RGB features in the spatial Transformer module. A channel-independent learning module is constructed based on a hierarchical multi-scale feature aggregation module, and the query vector of the RGB modality in the channel-independent learning module is represented by using the RGB image representation feature to obtain the query vector of the RGB modality represented in the channel-independent learning module; the key vector of the deep modality in the channel-independent learning module is represented by using the feature expression of the enhanced deep image to obtain the key vector of the deep modality represented in the channel-independent learning module; The query vector of the RGB modality represented in the channel-independent learning module and the key vector of the depth modality represented in the channel-independent learning module are sequentially reshaped and reversed to obtain the irrelevant weight matrix in the channel-independent learning module and the refined RGB features of the channel-independent learning module; The refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module are combined to obtain refined RGB features; the representation features of the multi-stage depth image and the feature expression of the enhanced RGB image are fused using a hierarchical multi-scale feature aggregation module to obtain refined depth features; The refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module are combined to obtain the spliced ​​refined RGB features; the spliced ​​refined RGB features are fused using the modal complementary feature retention module in the adaptive cross-modal fusion module to obtain the refined deep features; the refined deep features and the spliced ​​refined RGB features are sequentially spliced ​​and residual jumped using the residual feature fusion module in the adaptive cross-modal fusion module to obtain the fused cross-modal fusion features.

3. The RGB-D salient object detection method based on cross-modal association learning according to claim 2, characterized in that: In the step of extracting the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image by using the information encoding module to obtain the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image, the relationship between the corresponding processes is: ； in, Represents the representation features of multi-stage RGB images, Representation features of multi-stage depth images, The encoder backbone network represents the RGB image stream, The encoder backbone network represents the deep image stream, Indicates The channel index number of the stage feature map, Indicates The height of the stage feature map, Indicates The width of the stage feature map, represents an RGB image, represents the depth image, The dimension is The space of real number sets; In the step of inputting the representation features of the multi-stage RGB image and the representation features of the multi-stage depth image into the gated bottleneck component module for mechanism processing to obtain the feature attention weight map of the RGB image and the feature attention weight map of the depth image, the relationship between the corresponding processes is: ； in, Represents the feature attention weight map of the RGB image, Represents the feature attention weight map of the depth image, represents the normalization operation, represents the activation function, Indicates a convolution operation with a convolution kernel of 3x3. Representation and Different convolution kernels are 3x3 convolution operations; In the steps of fusing the feature attention weight map of the RGB image with the representation features of the multi-stage RGB image to obtain the enhanced feature expression of the RGB image; and fusing the feature attention weight map of the depth image with the representation features of the multi-stage depth image to obtain the enhanced feature expression of the depth image, the relationship between the corresponding processes is: ； in, represents the feature expression of the enhanced RGB image, represents the feature expression of the enhanced depth image, Represents an element-by-element multiplication operation; In the steps of performing a convolution operation on the representation features of the multi-stage RGB image to obtain a query vector of the RGB mode; and performing a global average pooling operation on the feature expression of the enhanced depth image to obtain a key vector of the depth mode, the corresponding process has the following relationship: ； in, represents the query vector in RGB modality, The key vector representing the depth modality, Indicates a convolution operation with a convolution kernel of 1x1. Representation and Different convolution kernels are 1x1 convolution operations. represents the global average pooling operation, The dimension is The space of real numbers, represents a convolutional layer with a kernel of 1x1 used to process the query vector, Represents a convolutional layer with a kernel of 1x1 used to process the key vector; In the step of building a spatial Transformer module based on the hierarchical multi-scale feature aggregation module, the query vector of the RGB modality and the key vector of the depth modality are calculated to obtain the spatial irrelevance mapping weight matrix between the RGB modality features and the depth modality features. The relationship between the corresponding process is: ； in, represents the spatially uncorrelated mapping weight matrix, represents a matrix tensor whose elements are all 1, represents the matrix multiplication operation, The dimension is The space of real number sets; In the step of using the spatially uncorrelated mapping weight matrix to perform element-by-element calculations to obtain the refined RGB features in the spatial Transformer module, the corresponding process has the following relationship: ； in, Represents the refined RGB features in the spatial Transformer module; In the step of constructing a channel-independent learning module based on a hierarchical multi-scale feature aggregation module, the query vector of the RGB modality in the channel-independent learning module is represented by using the RGB image representation feature to obtain the query vector of the RGB modality represented in the channel-independent learning module; and the key vector of the deep modality in the channel-independent learning module is represented by using the enhanced feature expression of the deep image to obtain the key vector of the deep modality represented in the channel-independent learning module. The relationship between the corresponding processes is: ； in, The query vector representing the RGB modality of the channel-independent learning module, The key vector representing the deep modality of the channel-independent learning module, represents the global average pooling operation, The dimension is The space of real number sets; In the steps of successively reshaping and reversely operating the query vector of the RGB modality represented in the channel-independent learning module and the key vector of the depth modality represented in the channel-independent learning module to obtain the irrelevant weight matrix in the channel-independent learning module and the refined RGB features of the channel-independent learning module, the corresponding process has the following relationship: ； in, represents the uncorrelated weight matrix between the query vector of the RGB modality and the key vector of the depth modality in the channel-independent learning module, represents a matrix tensor whose elements are all 1, represents a channel-by-channel multiplication operation, represents the refined RGB features in the channel-independent learning module, The dimension is The space of real number sets; In the step of combining the refined RGB features in the channel-independent learning module and the refined RGB features in the spatial Transformer module to obtain the concatenated refined RGB features, the relationship between the corresponding processes is: ； in, represents the refined RGB features, Represents a splicing operation; In the step of fusing the spliced ​​refined RGB features using the modal complementary feature retention module in the adaptive cross-modal fusion module to obtain the refined depth features, and sequentially splicing and performing residual jump operations on the refined depth features and the spliced ​​refined RGB features using the residual feature fusion module in the adaptive cross-modal fusion module to obtain the fused cross-modal fusion features, the corresponding process has the following relationship: ； in, represents the concatenated cross-modal fusion features, Represents the residual feature fusion module in The output characteristics of the stage, represents the element-by-element addition operation, represents the batch normalization operation, represents the residual processing module used to enrich feature information, represents the refined RGB features, Represents refined deep features.

4. The RGB-D salient object detection method based on cross-modal association learning according to claim 3, characterized in that: In step 3, the refined small-scale features are upsampled to obtain upsampled features of the refined small-scale features. The relationship between the processes is: ； in, represents the upsampled features of refined small-scale features, represents the Gelu activation function, represents a 1x1 convolutional layer, represents a bilinear upsampling operation with a scale factor of 2, Represents small-scale features; Indicates that after bilinear upsampling operation After processing, the features Indicates The index of the hierarchical multi-scale feature aggregation module, The index number indicating the fusion level; The refined large-scale features are fused with the refined small-scale features to obtain the fused features of the refined large-scale features and the refined small-scale features. The fused features of the refined large-scale features and the refined small-scale features are subjected to convolution layer calculation to obtain the convolution layer calculation results of the fused features of the refined large-scale features and the refined small-scale features. The convolution layer calculation results are processed with the activation function to obtain the attention map. The corresponding process has the following relationship: ； in, represents the attention map, Represents large-scale features; The upsampled features of the refined large-scale features and the refined small-scale features are respectively subjected to attention enhancement processing by element-by-element multiplication to obtain enhanced large-scale features and enhanced small-scale features. The relationship between the corresponding processes is: ； in, represents enhanced small-scale features, represents the enhanced large-scale features, represents element-wise addition operation, Represents element-by-element multiplication operation; The enhanced large-scale features and enhanced small-scale features are fused using the hierarchical multi-scale feature aggregation module to obtain the output fusion features of the hierarchical multi-scale feature aggregation module. The corresponding process has the following relationship: ； in, Indicates The hierarchical multi-scale feature aggregation module is The output fusion features of the layer, Represents a splicing operation, Represents a 3x3 convolutional layer.

5. The RGB-D salient object detection method based on cross-modal association learning according to claim 4, characterized in that: In step 4, the output fusion features of the hierarchical multi-scale feature aggregation module are used to predict the object boundary by using the upsampling layer and the convolution layer to obtain the features of the edge prediction map. The corresponding process has the following relationship: ； in, Represents edge features of half the size of the input image, represents the features of the edge prediction map of the same size as the input image, represents a 3×3 convolutional kernel convolution layer containing a batch normalization layer and a ReLU activation function. Represents the batch normalization layer and ReLU activation function in the convolutional layer.

6. The RGB-D salient object detection method based on cross-modal association learning according to claim 5, characterized in that: In step 5, the output fusion feature of the hierarchical multi-scale feature aggregation module is subjected to convolution layer calculation to obtain the saliency feature of the output fusion feature. The relationship between the corresponding process is: ； in, Indicates significant features; In the step of performing point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map, the relationship between the corresponding process is: ； in, Represents the final saliency prediction map.

7. The RGB-D salient object detection method based on cross-modal association learning according to claim 6, characterized in that: In step 6, the RGB-D data is input into the RGB-D salient object detection network model for training, and the features of the edge prediction map and the final saliency prediction map are combined to generate the RGB-D salient object detection result, which specifically includes the following steps: The dataset of RGB images, the dataset of saliency reference truth maps corresponding to RGB images, the dataset of depth images, and the dataset of saliency reference truth maps corresponding to depth images are made public, and the saliency reference truth maps corresponding to the RGB images in the dataset are used to perform element-by-element calculations to obtain salient target areas and non-salient areas of the RGB images; the saliency reference truth maps corresponding to the depth images in the dataset are used to perform element-by-element calculations to obtain salient target areas and non-salient areas of the depth images; Combining the salient target area and the non-salient area of ​​the RGB image and the final saliency prediction map to obtain a reconstructed RGB image; combining the salient target area and the non-salient area of ​​the depth image and the final saliency prediction map to obtain a reconstructed depth image; The structural similarity is used to measure the RGB image and the depth image to obtain the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image; The structural quality perception loss of the RGB image and the structural quality perception loss of the depth image are added to obtain the overall structural quality perception loss, and the saliency reference truth map is convolved with the Laplacian kernel to obtain the edge reference truth map; The edge reference true value map is combined with the features of the edge prediction map to calculate the edge prediction loss; The edge prediction loss and the overall structural quality perception loss are calculated to obtain the total loss for model optimization, and the total loss is used to optimize the model to generate RGB-D salient object detection results.

8. The RGB-D salient object detection method based on cross-modal association learning according to claim 7, characterized in that: In the step of using the saliency reference truth map corresponding to the RGB image in the data set to perform element-by-element calculation to obtain the salient target area and non-salient area of ​​the RGB image, the corresponding process has the following relationship: ； in, Indicates the salient target area, represents an RGB image, represents element-wise multiplication, represents the saliency reference truth map, represents the non-significant area, A binary inversion map representing the saliency reference truth map; In the step of measuring the RGB image and the depth image by using structural similarity to obtain the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image, the relationship between the corresponding processes is: ； in, represents the structural quality perceptual loss of RGB images, represents the structural quality-aware loss of the depth image, represents the reconstructed image of the RGB image, represents the structural similarity index measurement loss, A constant representing the importance of balancing losses, A constant representing the importance of balancing losses, represents the depth image, a reconstructed image representing the depth image; In the step of adding the structural quality perception loss of the RGB image and the structural quality perception loss of the depth image to obtain the overall structural quality perception loss, the relationship between the corresponding process is: ； in represents the perceived loss of overall structural quality; In the step of performing a convolution operation on the saliency reference truth map using the Laplacian kernel to obtain the edge reference truth map, the corresponding process has the following relationship: ； in, represents the edge reference truth map, represents the convolution operation, represents the Laplace kernel; In the step of calculating the edge prediction loss and the overall structural quality perception loss to obtain the total loss of model optimization, the relationship between the corresponding process is: ； in, represents the total loss of model optimization, represents a constant used to balance the significance loss and the marginal loss, represents a constant used to balance the significance loss and the marginal loss, represents the edge prediction loss, Represents ordinary mathematical multiplication between two numbers.

9. A cross-modal association learning RGB-D salient object detection system, characterized in that: The system applies the RGB-D salient object detection method of cross-modal association learning according to any one of claims 1 to 8, and the system comprises: Network building blocks for: A feature extraction backbone network is constructed based on a dual-stream encoder module and an adaptive cross-modal fusion module. A salient object decoder is constructed based on a hierarchical multi-scale feature aggregation module. The feature extraction backbone network and the salient object decoder constitute an RGB-D salient object detection network model. Feature extraction module for: The information encoding module in the dual-stream encoder module is used to extract the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image, and the scale representation features of the multi-stage RGB image and the scale representation features of the multi-stage depth image are hierarchically fused using an adaptive cross-modal fusion module to obtain fused cross-modal fusion features; Saliency calculation module, used to: The adjacent features of the fused cross-modal fusion features are represented to obtain adjacent scale features, and the adjacent scale features are fused using the fusion level of the hierarchical multi-scale feature aggregation module to obtain refined large-scale features and refined small-scale features; Performing upsampling processing on the refined small-scale features to obtain upsampled features of the refined small-scale features; The refined large-scale features are fused with the refined small-scale features to obtain fused features of the refined large-scale features and the refined small-scale features, a convolution layer calculation is performed on the fused features of the refined large-scale features and the refined small-scale features to obtain a convolution layer calculation result of the fused features of the refined large-scale features and the refined small-scale features, and the convolution layer calculation result is processed using an activation function to obtain an attention map; Using element-by-element multiplication, attention enhancement is performed on the upsampled features of the refined large-scale features and the refined small-scale features to obtain enhanced large-scale features and enhanced small-scale features; The enhanced large-scale features and the enhanced small-scale features are fused using a hierarchical multi-scale feature aggregation module to obtain output fusion features of the hierarchical multi-scale feature aggregation module; Edge-aware optimization module for: The output fusion features of the hierarchical multi-scale feature aggregation module are used to predict the object boundary using the upsampling layer and the convolution layer to obtain the features of the edge prediction map. Offline training module for: Perform convolution layer calculation on the output fusion features of the hierarchical multi-scale feature aggregation module to obtain the saliency features of the output fusion features, and perform point-by-point addition operation on the saliency features of the output fusion features combined with the features of the edge prediction map to obtain the final saliency prediction map; Salient object detection module, used to: The RGB-D data is input into the RGB-D salient object detection network model for training, and the features of the edge prediction map and the final saliency prediction map are combined to generate the RGB-D salient object detection results.

Images