RGBD saliency detection method based on multi-scale feature fusion
A technology of multi-scale features and detection methods, applied in the field of computer vision, can solve problems such as hindering RGBD images, interfering with effective expression of depth information, and limited depth image datasets.
Active Publication Date: 2020-06-05
HANGZHOU DIANZI UNIV
View PDF9 Cites 16 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Generally speaking, there are two main factors hindering the further development of RGBD image saliency detection tasks: First, although the emergence of devices such as Kinect and light field cameras has greatly facilitated the acquisition of depth information, it still inevitably introduces a large number of Noise, to a certain extent, interferes with the effective expression of depth information. At the same time, the existing depth image datasets available are extremely limited, lacking large-scale datasets such as the RGB image dataset ImageNet, and it is difficult to fit network models with complex structures. ; Second, how to effectively fuse the information of two different modalities, RGB information and depth information, is challenging. RGB images contain a lot of semantic information such as color and texture, while depth images contain rich edges and Geometric information such as shape, the two complement each other, which is conducive to more accurate highlighting of salient regions
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0017] The method of the present invention will be further described below in conjunction with the accompanying drawings.
[0018] Such as figure 1 As shown, the significance detection method of the present invention, the steps are as follows:
[0019] Step (1), building a saliency detection model.
[0020] The saliency detection model includes a two-stream feature extraction module, a multi-scale feature pooling module, a multi-scale feature aggregation module, a deep fusion module and a saliency boundary refinement module.
[0021] Step (2), processing the original depth image of the RGB image I through the HHA algorithm to obtain the depth image D.
[0022] Step (3), the RGB image I and its depth image D are input into the saliency detection model, and the multi-level RGB image feature {I i ,i=1,2,3,4} and depth image features {D i ,i=1,2,3,4}.
[0023] Step (4), further extracting deep-level features through the multi-scale feature pooling module and the multi-scale f...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides an RGBD saliency detection method based on multi-scale feature fusion. The method comprises the following steps of: firstly, constructing a saliency detection model, and extracting multi-stage RGB image features and depth image features through a double-flow feature extraction module, further extracting deep features through a multi-scale feature pooling module and a multi-scale feature aggregation module, and meanwhile, fusing the features from the feature extraction branches, the multi-scale feature pooling module and the multi-scale feature aggregation module step bystep by utilizing the deep fusion module. A saliency boundary refinement module performs boundary constraint through shallow features from an RGB image feature extraction branch and a depth image feature extraction branch to achieve the purpose of boundary refinement; and meanwhile, global constraint is carried out by utilizing output features of the deep fusion module, so that the purpose of global optimization is achieved. According to the method, end-to-end saliency prediction is realized, edge information is introduced into saliency detection, and RGB image information and depth image information can be fully and effectively utilized to predict a saliency region.
Description
technical field [0001] The invention belongs to the field of computer vision, and in particular uses a deep convolutional neural network to fuse feature information contained in RGB images and depth images through a multi-scale method. Background technique [0002] Saliency detection aims to distinguish the most visually distinct objects or regions in a scene, and has a wide range of applications in the fields of visual tracking, image segmentation, and object detection. At the same time, with the rapid development of deep learning technology, convolutional neural network has become the mainstream method for processing saliency detection tasks. However, most of the existing saliency detection methods based on deep learning are aimed at 2D image saliency detection tasks, that is, only relying on RGB images and ignoring the corresponding depth information, which greatly limits the accuracy and efficiency of saliency detection. , especially when salient objects are indistingui...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06K9/46G06K9/62
CPCG06V10/462G06F18/253
Inventor颜成钢温洪发周晓飞孙垚棋张继勇张勇东
OwnerHANGZHOU DIANZI UNIV