A depth quality weighted based RGB-D salient object detection method

Abstract

The present application belongs to the field of computer vision, and provides an RGB-D saliency object detection method based on depth quality weighting, comprising the following steps: 1) obtaining an RGB-D dataset for training and testing the task, and defining the algorithm target of the present application; 2) constructing an RGB encoder for extracting RGB image features and a depth (Depth) image feature encoder; 3) constructing a cross-modal weighted fusion module, and guiding the weighted fusion of the extracted RGB image features and Depth image features through a depth quality evaluation mechanism guided by a weighting formula; 4) constructing a bidirectional scale correlation convolution mechanism for multi-scale feature extraction and fusion, so as to enhance the advanced semantic information of multi-modal features; 5) establishing a decoder to generate a saliency map P est ; 6) calculating the loss of the predicted saliency map P est and the manually labeled saliency object segmentation map P GT ; 7) testing the test dataset to generate a saliency map P est , and performing performance evaluation using evaluation indexes. The present application can effectively integrate complementary information from different modal images, and improve the accuracy of saliency object prediction in complex scenes.

Description

Technical fields:

[0001] This invention relates to the fields of computer vision and image processing, and in particular to a depth-quality-weighted RGB-D salient object detection method. Background technology:

[0002] In the fields of computer vision and image processing, salient object detection (SOD) aims to identify and segment the most attractive objects or regions in given data (such as RGB images, RGB-D images, videos, etc.) by simulating human visual attention mechanisms. It has been widely used in various computer vision tasks, such as semantic segmentation, image compression, object tracking, and so on.

[0003] Single-modal RGB salient object detection algorithms struggle to locate salient objects from cluttered backgrounds, especially under challenging conditions such as complex backgrounds and lighting. One approach to overcome these challenges is to use depth maps to compensate for the spatial information missing in RGB images. While RGB images contain detailed information (such as rich texture, color, and appearance cues), depth maps provide spatial information, representing geometric structure and distance. Therefore, combining RGB images with depth maps for SOD tasks (called RGB-D SOD) is a reasonable choice, as it can handle more complex scenes and meet the needs of advanced detection.

[0004] While existing RGB-D SOD methods have made significant progress, most have overlooked the detrimental impact of low-quality depth maps on the RGB-D SOD task. High-quality depth maps, with their sharp boundaries and accurate target localization, contribute to SOD. However, low-quality depth maps not only have blurred edges but also inaccurate target localization, which can introduce noise into cross-modal feature fusion, thus degrading SOD performance. Therefore, it is necessary to consider the quality of the depth map in the RGB-D SOD task.

[0005] Considering that low-quality depth maps inevitably impact salient object detection, this invention explores an efficient cross-modal feature fusion method to effectively reduce the impact of low-quality depth maps on salient object detection. Furthermore, to further explore the complementary information between multi-scale features and fully utilize the advantages of high-level features and long-distance information association, this invention leverages the role of multi-scale feature fusion to help salient object detection models more accurately predict salient objects. Summary of the Invention:

[0006] To address the problems mentioned above, this invention provides a depth-quality-weighted RGB-D saliency target detection method, the specific technical solution of which is as follows:

[0007] 1. Obtain the RGB-D dataset for training and testing this task.

[0008] The NJUD, NLPR, and DUT-RGBD datasets were used as the training set, and the remaining parts of the NJUD, NLPR, SIP, LFSD, and RGBD135 datasets were used as the test set.

[0009] 2. Construct a salient object detection model network using convolutional neural networks to extract RGB image features and depth image features.

[0010] 2.1) VGG16 is used as the backbone network of the model in this invention to extract RGB image features and corresponding Depth image features, respectively. and Where i represents the layer number, corresponding to each layer output of VGG16.

[0011] 2.2) The VGG16 weights used in this invention for constructing the backbone network are initialized using VGG16 parameter weights pre-trained on the ImageNet dataset.

[0012] 3. Based on the multi-scale RGB image features extracted in step 2 and corresponding Depth image features Multi-scale cross-modal feature weighted fusion is performed, and this weighted fusion is used to construct a cross-modal feature fusion network for generating multimodal features.

[0013] 3.1) The cross-modal feature fusion network extracts RGB image features from five levels of cross-modal weighted fusion (CMWF) modules. and corresponding Depth image features It constructs and generates multimodal features at 5 levels.

[0014] 3.2) The input of the CMWF module at level i is composed of data. and It constitutes and generates multimodal features at level i through a weighted guided deep quality assessment mechanism.

[0015] 3.3) The CMWF module generates multimodal features through a weighted guided deep quality assessment mechanism. The specific process is as follows:

[0016] 3.3.1) First, this invention constructs a channel-spatial attention feature enhancement module to filter and enhance features, thereby improving the saliency of the features. Through this channel-spatial attention feature enhancement module, unnecessary noise can be further removed, and common salient objects can be emphasized, resulting in enhanced multimodal features.

[0017]

[0018]

[0019]

[0020] Where c∈{r,d}, and Represents channel attention and spatial attention at level i, GAP represents global average pooling, GMP represents global max pooling, Cat represents feature connection operation, and Conv... k This indicates a convolution operation with a kernel size of k×k, sigmoid represents the sigmoid activation function, and multi represents an element-aware matrix multiplication operation.

[0021] 3.3.2) The difference between the two modalities at the feature level is reflected by calculating the difference between the enhanced RGB feature-level attention map and the deep feature-level attention map. Then, the difference is divided by the absolute value of the enhanced RGB feature pixel value to obtain the weighting coefficient λ. i :

[0022]

[0023]

[0024] Where subtra represents element-aware matrix subtraction. |·| represents the average absolute operation, and H and W are the height and width of feature f.

[0025] 3.3.3) Further employing a cross-enhancement strategy, the original RGB features are... and depth features RGB image features after channel-spatial attention feature enhancement and corresponding Depth image features By employing a cross-enhancement strategy, cross-enhancement features are obtained. and

[0026]

[0027]

[0028] 3.3.4) After obtaining the weighted coefficients and cross-enhancement features, a weighted fusion method is used to fuse cross-modal features and RGB image features. and corresponding Depth image features Obtain fusion features

[0029]

[0030] Where i∈{1,2,3,4,5} represents the level of the model in which the feature is located, add represents the element-aware matrix addition operation, and Cat represents the feature concatenation operation.

[0031] 4) Through the above operations, multimodal features at 5 levels were extracted. The features from the 4th and 5th levels are then input into a bidirectional scale-correlated convolution module, which enhances the receptive field information and high-level semantic information of the multimodal features through depthwise separable convolution operations.

[0032] 4.1) Extract multi-scale receptive field information from the multimodal features of layers 4 and 5 using depthwise separable convolution operations, and set depthwise separable convolutions with different kernel sizes:

[0033] R1=DConv3(R)+R Formula (9)

[0034] R i =DConv 2×i+1 (R i-1 )+R,i∈(2,3,4) Formula (10)

[0035]

[0036] Where R represents the input features, DConv3 represents a 3×3 depthwise separable convolution, and DConv... 2×i+1 This indicates a depthwise separable convolution with a kernel of 2×i+1.

[0037] 4.2) Connect all the above multi-scale features together and add a residual connection to obtain high-level features.

[0038]

[0039] Where c∈{4,5}, and A represents global average pooling.

[0040] 4.3) The low-level features generated in the above steps and and high-level features and The input is fed into the decoder network to obtain the final fused features. After activation by the sigmoid function, the predicted saliency map P is obtained. est :

[0041]

[0042] 5) The saliency map P predicted by this invention est Compared with manually annotated salient target segmentation map P GT The loss function is calculated, and the parameter weights of the proposed model are updated step by step using the Adam and backpropagation algorithms to finally determine the structure and parameter weights of the RGB-D salient object detection algorithm.

[0043] 6) Based on the model structure and parameter weights determined in step 5, test the RGB-D image pairs on the test set to generate a saliency map P. test The evaluation was conducted using MAE, S-measure, F-measure, and E-measure metrics.

[0044] This invention relates to multimodal salient object detection based on deep convolutional neural networks. It utilizes the rich spatial structure information in depth images and employs a weighted, guided depth quality assessment mechanism for cross-modal feature fusion with depth features extracted from RGB images. This allows it to adapt to the requirements of salient object detection in various scenarios, particularly challenging ones (complex backgrounds, low contrast, transparent objects, etc.), where it demonstrates robustness. Compared to previous RGB-D salient object detection methods, the advantages of this invention are as follows:

[0045] First, deep learning technology is used to construct the relationship between RGB-D image pairs and salient targets in the images through an encoder and decoder structure. Saliency prediction is obtained through cross-modal feature extraction and fusion. Second, a weighted fusion method is used to effectively modulate the supplementary information of depth image features to RGB image features, and the inherent depth distribution information is used to guide cross-modal feature fusion, eliminating interference from background information in the RGB images and laying a foundation for the next stage of salient target prediction. Finally, the constructed decoder performs multi-scale, multi-modal feature fusion to predict the final saliency map. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the model structure of the present invention.

[0047] Figure 2 Schematic diagram of cross-modal feature fusion module

[0048] Figure 3A schematic diagram of a bidirectional scale-correlated convolutional module.

[0049] Figure 4 Diagram of a decoder

[0050] Figure 5 A schematic diagram of model training and testing.

[0051] Figure 6 This is a comparison chart of the results of this invention with other RGB-D saliency target detection methods. Detailed Implementation

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Furthermore, the described examples are merely a part of the examples of the present invention, and not all of them. All other examples obtained by those skilled in the art based on the examples in this invention without inventive effort are within the scope of protection of this invention.

[0053] Reference Appendix Figure 1 A depth-quality-weighted RGB-D saliency target detection method mainly includes the following steps:

[0054] 1. Obtain the RGB-D dataset for training and testing this task, define the algorithm objective of this invention, and determine the training and test sets for training and testing the algorithm. Use the NJUD dataset, NLPR dataset, and DUT-RGBD dataset as the training set, and the remaining datasets as the test set, including the remaining parts of the NJUD dataset, the remaining parts of the NLPR dataset, the SIP dataset, the LFSD dataset, and the RGBD135 dataset.

[0055] 2. Construct a salient object detection model network using a convolutional neural network for extracting RGB image features and depth image features, including an RGB encoder for extracting RGB image features and a depth encoder for extracting depth image features:

[0056] 2.1. Input the RGB image with three channels into the RGB encoder to generate RGB image features at 5 levels, namely...

[0057] 2.2. The three-channel depth image is input into the depth encoder to generate five levels of depth image features, namely...

[0058] 3. Reference Appendix Figure 2 As shown, the RGB image features of the five levels generated in step 2 are combined through a cross-modal fusion module. and Depth image features Weighted fusion was performed to obtain multimodal features at five levels. The main steps are as follows:

[0059] 3.1. The cross-modal feature fusion network consists of a 5-level cross-modal weighted fusion CMWF module, extracting RGB image features from 5 levels. and corresponding Depth image features It constructs and generates multimodal features at 5 levels.

[0060] 3.2. The input data for the CMWF module at level i is: and It constitutes and outputs the multimodal features of the i-th level through a weighted guided deep quality assessment mechanism.

[0061] 3.3. The CMWF module generates multimodal features through a weighted guided deep quality assessment mechanism. The specific process is as follows:

[0062] 3.3.1. First, this invention constructs a channel-spatial attention feature enhancement module to filter and enhance features, thereby improving their saliency. Through this module, unnecessary noise can be further removed, and common salient objects can be emphasized, resulting in enhanced multimodal features.

[0063]

[0064]

[0065]

[0066] Where c∈{r,d}, and Represents channel attention and spatial attention at level i, GAP represents global average pooling, GMP represents global max pooling, Cat represents feature connection operation, and Conv... k This represents a convolution operation with a kernel size of k×k, sigmoid represents the sigmoid activation function, and multi represents an element-aware matrix multiplication operation.

[0067] 3.3.2. The difference between the two modalities at the feature level is reflected by calculating the difference between the enhanced RGB feature-level attention map and the deep feature-level attention map. Then, the difference is divided by the absolute value of the enhanced RGB feature pixel value to obtain the weighting coefficient λ. i :

[0068]

[0069]

[0070] Where subtra represents element-aware matrix subtraction. |·| represents the average absolute operation, and H and W are the height and width of feature f.

[0071] 3.3.3. Further employing a cross-enhancement strategy, we will use the original RGB features... and depth features RGB image features after channel-spatial attention feature enhancement and corresponding Depth image features By employing a cross-enhancement strategy, cross-enhancement features are obtained. and

[0072]

[0073]

[0074] 3.3.4. After obtaining the weighted coefficients and cross-enhancement features, a weighted fusion method is used to fuse cross-modal features and RGB image features. and corresponding Depth image features Obtain fusion features

[0075]

[0076] Where i∈{1,2,3,4,5} represents the level of the model in which the feature is located, add represents the element-aware matrix addition operation, and Cat represents the feature concatenation operation.

[0077] 4. Reference Appendix Figure 3 The receptive field information and high-level semantic information of multimodal features are enhanced by using bidirectional scale-correlated convolutional modules:

[0078] 4.1) Extract multi-scale receptive field information from the multimodal features of layers 4 and 5 using depthwise separable convolution operations, and set depthwise separable convolutions with different kernel sizes:

[0079] R1=DConv3(R)+R Formula (9)

[0080] R i =DConv 2×i+1 (R i-1 )+R,i∈(2,3,4) Formula (10)

[0081]

[0082] Where R represents the input features, DCconv3 represents a 3×3 depthwise separable convolution, and DConv 2×i+1 This indicates a depthwise separable convolution with a kernel of 2×i+1.

[0083] 4.2) Connect all the above multi-scale features together, and add a residual connection to stabilize and optimize the high-level features.

[0084]

[0085] Where c∈{4,5}, and A represents global average pooling.

[0086] 5. Reference Appendix Figure 4 The acquired low-level features and and high-level features and The input is fed into the decoder network and then activated by the sigmoid function to obtain the predicted saliency map P. est :

[0087]

[0088] 6) The saliency map P predicted by this invention est Compared with manually annotated salient target segmentation map P GT The loss function is calculated, and the parameter weights of the proposed model are updated step by step using the Adam and backpropagation algorithms to finally determine the structure and parameter weights of the RGB-D saliency detection algorithm.

[0089] 7) Based on the model structure and parameter weights determined in step 6, test the RGB-D image pairs on the test set to generate a saliency map P. test The evaluation was conducted using MAE, S-measure, F-measure, and E-measure metrics.

[0090] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. Any modifications or equivalent substitutions made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A depth-quality-weighted RGB-D saliency target detection method, characterized in that, The method includes the following steps: 1) Obtain the RGB-D dataset for training and testing the salient object detection task, define the algorithmic objective of this method, and determine the training and testing sets for training and testing the algorithm; 2) Construct an RGB encoder for extracting RGB image features and a Depth encoder for extracting depth image features; 3) Establish a cross-modal feature fusion network, and guide the cross-weighted fusion of RGB image features and Depth image features through a weighted guided deep quality assessment mechanism; 4) Based on the multimodal features fused from the above cross-modal characteristics, a bidirectional scale-related convolutional fusion mechanism is constructed to enhance the high-level semantic information of the multimodal features. Specifically, multi-scale receptive field information is extracted from the multimodal features at levels 4 and 5 through depthwise separable convolution operations. Depthwise separable convolutions with kernel sizes of 3×3, 5×5, 7×7, and 9×9 are set, and multi-scale features are fused through residual connections. Subsequently, the multi-scale features are concatenated and compressed through 1×1 convolutions, while residual connections are added for stable optimization to generate high-level features. 5) Build a decoder by inputting the first three low-level multimodal features and the high-level multiscale complementary features generated in step 4 into the decoder, and obtain the final predicted saliency map through the sigmoid activation function. 6) The predicted saliency map P est Compared with manually annotated salient target segmentation map P GT The loss function is calculated, and the proposed model parameter weights are updated step by step using the Adam and backpropagation algorithms to finally determine the structure and parameter weights of the RGB-D saliency detection algorithm. 7) Based on the model structure and parameter weights determined in step 6, test the RGB-D image pairs on the test set to generate a saliency map P. test And use evaluation metrics to assess performance.

2. The RGB-D saliency target detection method based on depth quality weighting according to claim 1, characterized in that: The specific method for step 1) is as follows: The NJUD, NLPR, and DUT-RGBD datasets were used as the training set, and the remaining parts of the NJUD, NLPR, SIP, LFSD, and RGBD135 datasets were used as the test set.

3. The RGB-D saliency target detection method based on depth quality weighting according to claim 1, characterized in that: The specific method for step 2) is as follows: 2.1) VGG16 is used as the backbone network of the model to extract RGB image features and corresponding Depth image features, respectively. and Where i represents the layer number, corresponding to the output of each layer of VGG16; 2.2) The backbone network is a VGG16 structure, and its weights are initialized by a VGG16 model pre-trained on the ImageNet dataset.

4. The RGB-D saliency target detection method based on depth quality weighting according to claim 1, characterized in that: The specific method for step 3) is as follows: 3.1) The cross-modal weighted fusion network consists of five layers of cross-modal weighted fusion CMWF modules, generating five layers of multimodal features. , 3.2) The input data of the CMWF module at level i is provided by and It constitutes and generates multimodal features at level i through a weighted guided deep quality assessment mechanism.

5. The RGB-D saliency target detection method based on depth quality weighting according to claim 1, characterized in that: The specific method for step 4) is as follows: 4.1) Extract multi-scale receptive field information from the multimodal features of layers 4 and 5 using depthwise separable convolution operations, and set depthwise separable convolutions with different kernel sizes: R1=DConv3(R)+R Formula (9) R i =DConv 2×i+1 (R i-1 )+R,i∈(2,3,4) Formula (10) Where R represents the input features, DConv3 represents a 3×3 depthwise separable convolution, and DConv... 2×i+1 This indicates a depthwise separable convolution with a kernel of 2×i+1; 4.2) Connect all the above multi-scale features together, and add a residual connection to stabilize and optimize the high-level features. 4.3) The first three low-level multimodal features obtained in step 4 are used to... and And the two high-level multi-scale complementary features obtained in step 5 and The input is fed into the decoder to obtain the final fused features. After activation by the sigmoid function, the predicted saliency map P is obtained. est : .

Images