An RGB-D salient object detection method based on multi-level feature and context information fusion

Abstract

The application belongs to the field of computer vision, and discloses an RGB-D saliency object detection method based on multi-level feature and context information fusion, which comprises the following steps: (1) taking VGG-16 as a main framework to extract multi-scale information from RGB images and depth images and then encoding; (2) using a proposed detail enhancement fusion module and a semantic enhancement fusion module to fuse the detail information in low-level features and the semantic information in high-level features respectively; (3) using a multi-scale receptive field enhancement module to obtain rich context information in the network; (4) constructing a densely connected feature pyramid decoder, inputting the features into the decoder to generate a saliency result; and (5) using binary cross entropy to train the model of the application to calculate the error between the predicted value and the true value at different pixels.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing, and in particular to an RGB-D salient target detection method based on the fusion of multi-level features and contextual information. Background Technology

[0002] In recent years, with the rapid development of the internet and the increasing prevalence of mobile devices such as smartphones, language and text are no longer sufficient to express the richness of information. Images and videos have thus entered our lives in large quantities and are being accessed by more and more people. While massive amounts of image data have provided convenience to our lives, information redundancy has also followed. Humans are naturally sensitive to images and can quickly and accurately identify the most intuitive foreground objects in a scene, i.e., salient targets. However, identifying salient targets in images is very challenging for computers. If we can effectively separate salient targets from background areas and prioritize the allocation of limited computing resources for subsequent image processing, it will greatly alleviate the problem of limited computer resources.

[0003] Inspired by the human visual attention mechanism, salient object detection aims to identify the most attractive objects or regions from input images in real-world scenes. As a preprocessing technique, salient object detection has been widely applied to many related computer vision tasks, such as object detection, semantic segmentation, visual question solving, video tracking, and face recognition. However, when objects and backgrounds have similar appearances, RGB image-based salient object detection results are unsatisfactory. With the advent of affordable portable devices (such as depth cameras), depth information has become readily available, which can provide supplementary information to RGB features, such as spatial structure, 3D distribution, and object edges. Therefore, many works have attempted to integrate depth maps into RGB salient object detection, i.e., RGB-D salient object detection.

[0004] Currently, many RGB-D salient object detection methods employ the same approach to fuse RGB and depth features at different levels. However, lower-level features contain more spatial details, while higher-level features contain rich semantic information. Furthermore, multi-scale contextual semantic information is crucial for efficiently locating salient objects and completely segmenting them from the background. To address these issues, this invention designs an RGB-D salient object detection method based on the fusion of multi-level features and contextual information to efficiently utilize features and contextual semantic information at different levels. Specifically, a detail enhancement fusion module (DEFM) and a semantic enhancement fusion module are designed to fuse multimodal information at different levels and extract detailed and semantic features more effectively through different combinations of channel attention and spatial attention. In addition, a multi-scale receptive field enhancement module is designed, which increases the receptive field through parallel operations of dilated convolutions with different kernel sizes and different padding rates. Skip connections are then added in each stream to make pixel sampling denser. Therefore, this model can effectively integrate contextual information during feature extraction. Summary of the Invention

[0005] To address the problems mentioned above, this invention provides an RGB-D salient target detection method based on the fusion of multi-level features and contextual information. The specific technical solution adopted is as follows:

[0006] 1. Using VGG-16 as the backbone, multi-scale information is extracted from RGB and depth images and then encoded.

[0007] 1.1) Randomly select 700 samples from the NLPR dataset, 1485 samples from the NJU2K dataset, and 800 samples from the DUT dataset as the training set, and classify the remaining samples from the first three datasets, as well as the samples from the RGBD135 and STERE datasets, as the test set.

[0008] 1.2) The pre-trained VGG-16 is used as the backbone to extract features from the input RGB and depth images, and then the features are encoded.

[0009] 2. The proposed detail enhancement fusion module and semantic enhancement fusion module are used to fuse the detail information in low-level features and the semantic information in high-level features, respectively;

[0010] 2.1) In order to extract more effective features at each layer for salient object detection, a detail enhancement fusion module (DEFM) is proposed to fuse the features of the first two layers to better refine the salient boundaries, and a semantic enhancement fusion module (SEFM) is used to fuse the features of the last three layers to roughly locate salient objects.

[0011] 2.2) The features of the first two layers are fused by the detail enhancement fusion module, which uses channel attention and spatial attention mechanisms to generate enhanced features. Then, the two enhanced features are concatenated element by element to generate new fused features, as described below:

[0012]

[0013]

[0014]

[0015] in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This indicates a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. i+1 This represents the output feature from the (i+1)th fusion module.

[0016] 2.3) The features of the last three layers are fused by the semantic enhancement fusion module, as described below:

[0017]

[0018]

[0019]

[0020] in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This indicates a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. i+1 This represents the output feature from the (i+1)th fusion module.

[0021] 3. Use multi-scale receptive field enhancement modules to obtain rich contextual information in the network;

[0022] 3.1) Construct a multi-scale receptive field enhancement module, which increases the receptive field by parallel operation of convolutional kernels of different sizes and dilated convolutions with different filling rates, so as to better obtain contextual semantic information;

[0023] 3.2) First, the fused features generated by DEFM and SEFM in step (2) are input into the receptive field enhancement module. First, 1×1 convolutions are used to reduce the computational cost and complexity of the network model. Then, the input features are used to generate new features through 3×3, 5×5, and 7×7 convolution kernels, respectively. The specific process is as follows:

[0024]

[0025]

[0026] Where CA(·) represents the channel attention mechanism, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolution operations with kernels of 3×3, 5×5, and 7×7, respectively.

[0027] 3.3) Then, the new features are input into dilated convolutions with 3×3 kernels and padding ratios of 1, 2, and 4 to obtain multi-scale features Z1, Z2, and Z3. Finally, the obtained multi-scale features are concatenated to obtain new features. The description is as follows:

[0028]

[0029]

[0030] Conv1, Conv2, and Conv4 represent 3×3 convolutions with padding rates of 1, 2, and 4, respectively.

[0031] 4. Construct a densely connected feature pyramid decoder, and input these features into the decoder to generate saliency results;

[0032] 4.1) Input the multimodal features obtained in step 3 into the decoder composed of a dense feature pyramid network to obtain the final fused features. After activation by the sigmoid function, the predicted saliency map is obtained.

[0033] The advantages of this invention are as follows: Considering the differences between features at different levels, this invention proposes a detail enhancement fusion module and a semantic enhancement fusion module to better fuse more detailed information from low-level features and more semantic information from high-level features. Simultaneously, to better capture contextual information, dilated convolutions with different padding rates are used to increase the network's receptive field. Subsequently, a dense feature pyramid decoder is constructed to perform multimodal feature fusion and produce the final salient result. The proposed method achieves better results compared to existing RGB-D salient object detection methods. Attached Figure Description

[0034] Figure 1 Overall network flowchart

[0035] Figure 2 Detail Enhancement Fusion Module Diagram

[0036] Figure 3 Schematic diagram of semantic enhancement fusion module

[0037] Figure 4Schematic diagram of multi-scale receptive field enhancement module

[0038] Figure 5 Qualitative comparison chart with current best-performing methods Detailed Implementation

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Furthermore, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] Reference Appendix Figure 1 A method for RGB-D salient object detection based on the fusion of multi-level features and contextual information mainly includes the following steps:

[0041] 1. Using VGG-16 as the backbone, multi-scale information is extracted from RGB and depth images and then encoded.

[0042] 1.1) Randomly select 700 samples from the NLPR dataset, 1485 samples from the NJU2K dataset, and 800 samples from the DUT dataset as the training set, and classify the remaining samples from the first three datasets, as well as the samples from the RGBD135 and STERE datasets, as the test set.

[0043] 1.2) The pre-trained VGG-16 is used as the backbone to extract features from the input RGB and depth images, and then the features are encoded.

[0044] 2. The proposed detail enhancement fusion module and semantic enhancement fusion module are used to fuse the detail information in low-level features and the semantic information in high-level features, respectively;

[0045] 2.1) In order to extract more effective features at each layer for salient object detection, a detail enhancement fusion module (DEFM) is proposed to fuse the features of the first two layers to better refine the salient boundaries, and a semantic enhancement fusion module (SEFM) is used to fuse the features of the last three layers to roughly locate salient objects.

[0046] 2.2) The features of the first two layers are fused by the detail enhancement fusion module, which uses channel attention and spatial attention mechanisms to generate enhanced features. Then, the two enhanced features are concatenated element by element to generate new fused features, as described below:

[0047]

[0048]

[0049]

[0050] in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This indicates a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. i+1 This represents the output feature from the (i+1)th fusion module.

[0051] 2.3) The features of the last three layers are fused by the semantic enhancement fusion module, as described below:

[0052]

[0053]

[0054]

[0055] in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This indicates a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. i+1 This represents the output feature from the (i+1)th fusion module.

[0056] 3. Use multi-scale receptive field enhancement modules to obtain rich contextual information in the network;

[0057] 3.1) Construct a multi-scale receptive field enhancement module, which increases the receptive field by parallel operation of convolutional kernels of different sizes and dilated convolutions with different filling rates, so as to better obtain contextual semantic information;

[0058] 3.2) First, the fused features generated by DEFM and SEFM in step (2) are input into the receptive field enhancement module. First, 1×1 convolutions are used to reduce the computational cost and complexity of the network model. Then, the input features are used to generate new features through 3×3, 5×5, and 7×7 convolution kernels, respectively. The specific process is as follows:

[0059]

[0060]

[0061] Where CA(·) represents the channel attention mechanism, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolution operations with kernels of 3×3, 5×5, and 7×7, respectively.

[0062] 3.3) Then, the new features are input into dilated convolutions with 3×3 kernels and padding ratios of 1, 2, and 4 to obtain multi-scale features Z1, Z2, and Z3. Finally, the obtained multi-scale features are concatenated to obtain the new feature E. r i gbd The description is as follows:

[0063]

[0064]

[0065] Conv1, Conv2, and Conv4 represent 3×3 convolutions with padding rates of 1, 2, and 4, respectively.

[0066] 4. Construct a densely connected feature pyramid decoder, and input these features into the decoder to generate saliency results;

[0067] 4.1) Input the multimodal features obtained in step 3 into the decoder composed of a dense feature pyramid network to obtain the final fused features. After activation by the sigmoid function, the predicted saliency map is obtained.

[0068] 5. To demonstrate the efficiency of our proposed RGB-D salient object detection method based on the fusion of multi-level features and contextual information, such as... Figure 5 As shown, we present the saliency maps of the network predictions. The first column is the RGB map, the second column is the depth map, the third column is the ground truth (GT) map, the fourth to seventh columns are the saliency maps displayed by the current best methods, and the eighth column is the saliency map we are showing.

[0069] The above description is for preferred embodiments of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for RGB-D salient target detection based on the fusion of multi-level features and contextual information, characterized in that, The method includes the following steps: (1) VGG-16 is used as the backbone to extract multi-scale information from RGB and depth images and then encode it; (2) The proposed detail enhancement fusion module and semantic enhancement fusion module are used to fuse the detail information in the low-level features and the semantic information in the high-level features, respectively; The specific method of step (2) is as follows: 2.1) In order to extract more effective features at each layer for salient object detection, a detail enhancement fusion module (DEFM) is proposed to fuse the features of the first two layers to better refine the salient boundaries, and a semantic enhancement fusion module (SEFM) is used to fuse the features of the last three layers to roughly locate salient objects. 2.2) The features of the first two layers are fused by the detail enhancement fusion module, which uses channel attention and spatial attention mechanisms to generate enhanced features. Then, the two enhanced features are concatenated element by element to generate new fused features, as described below: in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This represents a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. F i+1 This represents the output feature from the (i+1)th fusion module; 2.3) The features of the last three layers are fused by the semantic enhancement fusion module, as described below: in and Representing the i-th layer features from RGB and depth flows respectively, Conv 3×3 This represents a convolution operation with a 3×3 kernel. CA(·) and SA(·) represent channel attention and spatial attention, respectively. F i+1 This represents the output feature from the (i+1)th fusion module; (3) Use a multi-scale receptive field enhancement module to obtain rich contextual information in the network; The specific method of step (3) is as follows: 3.1) Construct a multi-scale receptive field enhancement module, which increases the receptive field by parallel operation of convolutional kernels of different sizes and dilated convolutions with different filling rates, so as to better obtain contextual semantic information; 3.2) First, the fused features generated by DEFM and SEFM in step (2) are input into the receptive field enhancement module. First, 1×1 convolutions are used to reduce the computational cost and complexity of the network model. Then, the input features are used to generate new features through 3×3, 5×5, and 7×7 convolution kernels, respectively. The specific process is as follows: Where CA(·) represents the channel attention mechanism, Conv 3×3 Conv 5×5 Conv 7×7 These represent convolution operations with kernels of 3×3, 5×5, and 7×7, respectively. 3.3) Then, the new features are input into dilated convolutions with 3×3 kernels and padding ratios of 1, 2, and 4 to obtain multi-scale features Z1, Z2, and Z3. Finally, the obtained multi-scale features are concatenated to obtain new features. The description is as follows: Where Conv1, Conv2, and Conv4 represent 3×3 convolutions with padding rates of 1, 2, and 4, respectively; (4) Construct a densely connected feature pyramid decoder and input these features into the decoder to generate saliency results.

2. The RGB-D salient target detection method based on multi-level feature and contextual information fusion according to claim 1, characterized in that: The specific method of step (1) is as follows: 1.1) Randomly select 700 samples from the NLPR dataset, 1485 samples from the NJU2K dataset, and 800 samples from the DUT dataset as the training set, and classify the remaining samples from the first three datasets, as well as the samples from the RGBD135 and STERE datasets, as the test set. 1.2) The pre-trained VGG-16 is used as the backbone to extract features from the input RGB and depth images, and then the features are encoded.

3. Input the multimodal features obtained in step (3) into the decoder composed of a dense feature pyramid network to obtain the final fused features. After activation by the sigmoid function, the predicted saliency map is obtained.

Images