A saliency detection method based on color image and depth image

Abstract

The present invention belongs to the technical field of image saliency detection, and is a saliency detection method based on color images and depth images. The color image and the depth image are sent to a dual-channel VGG-16 encoder for encoding. During the encoding process, an information interaction module is used to transmit information from different modalities; the deepest multimodal features of the dual-channel VGG-16 are sent to a cross-modal mutual guidance module for updating and fusion; the multi-level multimodal features obtained in the above encoding process are sent to a residual multi-scale aggregation module to realize multi-level feature residual aggregation, and the aggregated features use a feature pyramid network structure to realize the final progressive fusion of high-level to low-level decoding features, and output saliency detection results. The present invention makes good use of the complementary information of color images and depth images, can handle low-contrast and complex background situations that have been difficult to handle in saliency detection, and can be well applied to general images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image saliency detection, and relates to a saliency detection method based on color images and depth images, which can detect salient targets in any image and involves relevant knowledge of image processing. Background Art

[0002] With the continuous development of image processing technology, image saliency detection has been widely used as an image preprocessing method in many fields such as image compression, image classification and image segmentation.

[0003] In recent years, with the development of deep learning technology, saliency detection technology based on fully convolutional networks has made great progress in performance. For example, Wang et al. proposed a recurrent fully convolutional network in the 2016 ECCV conference's 'Saliency Detection with Recurrent Fully Convolutional Networks', which can predict saliency results end-to-end while introducing a recurrent mechanism to fix errors in the initial prediction. Hou et al. published 'Deeply supervised salient object detection with short connections' in the PAMI journal in 2019. The paper introduced a short connection mechanism in the multi-scale feature fusion at the decoding end, and fused semantic information at different levels in the form of skip layers, thereby enhancing network performance. Pang et al. designed a multi-scale interactive network in the CVPR 2020 conference's paper 'Multi-scale Interactive Network for Salient Object Detection' to fully capture the semantic information of different levels contained in the image.

[0004] Although these methods have achieved good results, they all use color images as input, and still cannot achieve satisfactory results when faced with low-contrast or cluttered scenes. Some recent methods have introduced depth images as supplementary information to assist in saliency prediction. Depth images provide an intuitive expression of spatial structure and provide good prior information. Piao et al. published the 'Depth-induced Multi-scale Recurrent Attention Network for Saliency Detection' at the 2019 ICCV conference, which uses the network features of depth images to generate attention weights and uses the weights to recalibrate the features generated from color images. Zhao et al. published the 'A Single Stream Network for Robust and Real-time RGB-D SalientObject Detection' at the 2020 ECCV conference, which directly uses depth images as attention weights to optimize the features of color images, which is simpler and more efficient. Although these methods have achieved good results, there are still some problems that need to be solved. First, the above methods all implement multimodal interaction in the decoder part, and they focus on the interaction of mid-level and high-level features, and the feature interaction at each level is independent, which ignores the correlation between features at different levels and leads to divergence between shallow and deep features. Secondly, previous saliency detection methods lack in-depth thinking on multi-level feature fusion at the decoding end. Simple progressive upsampling introduces some unnecessary interference information and lacks a reasonable guidance mechanism to filter them out. Summary of the invention

[0005] The technical problem to be solved by the present invention is: for any given color image and its corresponding depth image, the salient target of the image can be predicted through the network. And the invention can process complex salient images and show good characteristics when facing low contrast and cluttered backgrounds. In addition, since the cross-modal fusion structure designed in the encoder part of the invention adopts the residual mode, the invention can also achieve good saliency prediction even in the absence of a corresponding depth map.

[0006] The technical solution of the present invention is as follows:

[0007] A saliency detection method based on color image and depth image, the specific steps are as follows:

[0008] (1) Interactive encoding of multimodal deep features

[0009] The deep feature encoder VGG-16 can extract features of different levels of the input image well. It is widely used in the fields of saliency detection, image segmentation, etc. The present invention intends to use dual-path VGG-16 to extract multi-level deep features of color images and depth images respectively. Most importantly, the present invention makes full use of the sequential encoding characteristics of VGG-16 and inserts an information interaction module between the features of different modes at each level. The steps are as follows:

[0010]

[0011] The features of the i-th level after interaction and Then they are sent to VGG-16 to complete the next level of feature encoding. After this process, VGG-16, which originally only encodes color images or depth images, incorporates information from another modality. This process also connects two VGG-16s that originally did not want to work together.

[0012] (2) The deepest high-level features guide each other

[0013] For the deep features encoded by the dual-path VGG-16 in step (1), the present invention adopts a cross-modal mutual guidance module to combine the deepest features and and Enter the following formula:

[0014]

[0015] To complete the further deep interaction of the highest level multimodal features. The deepest feature package extracts the high-level semantic information of the image, which can express the global content of the image. The multimodal features are further updated through the above process to strengthen the features. and Consistency in expression, more consistent expression, can help the network accurately locate the most salient areas in the image.

[0016] (3) Progressive fusion of multi-level features

[0017] After steps (1) and (2), the present invention completes the multi-level feature encoding and multi-modal information transmission process of color images and depth images. Finally, the dual-path VGG-16 encoder outputs 5 levels of multi-modal features after deep interleaving, which are respectively Next, these five levels of features are first convolved by 1×1 convolution to reduce the dimension of the features and reduce the amount of calculation to obtain the initial features of the decoder These features are then fed into the residual multi-scale feature aggregation module:

[0018]

[0019] To realize the residual fusion and updating of multi-level features.

[0020] Next, the multi-scale encoder features interacted by the residual multi-scale feature aggregation module are fed into the feature pyramid network to complete the progressive fusion from high level to low level.

[0021] The beneficial effects of the invention are as follows: it is possible to integrate multimodal information from color images and depth images well, and obtain more accurate detection results. And because the information interaction module in step (1) is in the form of a residual, the dual-path encoder in step (1) can still operate normally when the depth image information is shielded. The invention can be well extended to other fields of image processing, such as image segmentation, image classification, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the system block diagram.

[0023] Figure 2 It is a structural block diagram of the cross-modal mutual guidance module.

[0024] Figure 3 (a) is the original color image, (b) is the original depth image, and (c) is the obtained saliency prediction map. DETAILED DESCRIPTION

[0025] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0026] A saliency detection method based on color image and depth image, the steps are as follows:

[0027] Step 1: Send the given color image and depth image to different VGG-16 encoders for feature encoding of different modal information; specifically: Figure 3 (a) Color image and Figure 3 (b) The depth image is input to Figure 1 The dual-path VGG-16 encoder shown in the figure extracts multi-level multimodal features through VGG-16 and information interaction modules.

[0028] Step 2: Each VGG-16 encoder contains five scale features. During the encoding process, the features of the color image at each scale and the features of the depth image The feature information between different modalities is fused and transmitted through the information interaction module. The specific formula is as follows:

[0029]

[0030] Where W1, W2, b1 and b2 represent learnable convolution parameters, and [·,·] represents feature parallel connection. The above formula first fuses features from different images through feature parallel connection and convolution operation, and the fused multimodal features transmit information to each branch of the dual-path encoder through addition operation.

[0031] Step 3: The deepest features after encoding the VGG-16 encoder and The cross-modal mutual guidance module is further used to promote the mutual guidance between the deepest features. The specific formula is as follows:

[0032]

[0033] in, Represents matrix multiplication; this module first adds the deepest features of the dual encoder and then learns the added features With the deepest features and The affinity matrix A between r and A d , then use the learned affinity matrix to update the deepest features, and the updated features and The final fusion is completed through the addition operation; at this point, the network encoding process is completed;

[0034] Step 4: Mix the multimodal features of the first four scales extracted by the encoder E5 first uses 1×1 convolution to reduce the dimension of the feature to generate the decoder initial feature These features use the residual multi-scale feature aggregation module to complete multi-scale feature aggregation and obtain features D1 to D5. The specific formula is as follows:

[0035]

[0036] Among them, Cat(·,·) represents feature parallel connection, and Conv(·) represents convolution operation. This module gradually completes the fusion from the highest level feature D5 to the lowest level feature D1 in a residual manner.

[0037] Step 5: The updated D1~D5 in step 4 are fused and up-sampled in a progressive manner similar to the feature pyramid network structure to complete the final multi-scale feature fusion and generate the final saliency detection result.

Claims

1. A saliency detection method based on color image and depth image, characterized in that: Here are the steps: Step 1: Send the given color image and depth image to different VGG-16 encoders for feature encoding of different modal information; Step 2: Each VGG-16 encoder contains five scale features. During the encoding process, the features of the color image at each scale and the features of the depth image The feature information between different modalities is fused and transmitted through the information interaction module. The specific formula is as follows: Where W1, W2, b1 and b2 represent learnable convolution parameters, and [·,·] represents feature parallel connection. The above formula first fuses features from different images through feature parallel connection and convolution operation, and the fused multimodal features transmit information to each branch of the dual-path encoder through addition operation. Step 3: The deepest features after encoding the VGG-16 encoder and The cross-modal mutual guidance module is further used to promote the mutual guidance between the deepest features. The specific formula is as follows: in, Represents matrix multiplication; this module first adds the deepest features of the dual encoder and then learns the added features With the deepest features and The affinity matrix A between r and A d , then use the learned affinity matrix to update the deepest features, and the updated features and The final fusion is completed through the addition operation; at this point, the network encoding process is completed; Step 4: Mix the multimodal features of the first four scales extracted by the encoder E5 first uses 1×1 convolution to reduce the dimension of the feature to generate the decoder initial feature These features use the residual multi-scale feature aggregation module to complete multi-scale feature aggregation. The specific formula is as follows: Among them, Cat(·,·) represents feature parallel connection, and Conv(·) represents convolution operation. This module gradually completes the fusion from the highest level feature D5 to the lowest level feature D1 in a residual manner. Step 5: The updated D1~D5 in step 4 are gradually fused and up-sampled to complete the final multi-scale feature fusion and generate the final saliency detection result.

Images