Asymmetric multi-modal fusion saliency detection method based on attention mechanism

Abstract

The invention discloses an asymmetric multi-modal fusion saliency detection method based on an attention mechanism. The method comprises: inputting an RGB image and a depth image of an original three-dimensional image into a convolutional neural network for training to obtain a corresponding saliency detection image; calculating a loss function between a set formed by the saliency detection imagegenerated by the model and a set formed by corresponding real human eye gaze images to obtain an optimal weight vector and an offset term of the convolutional neural network classification training model; and inputting the three-dimensional image in the selected data set into the trained convolutional neural network model to obtain a saliency detection image. According to the saliency detection method, RGB and depth map features are fully extracted by adopting an asymmetric coding structure, rich image information of RGB is effectively utilized after an internal sensing module is added, channel and spatial attention mechanisms are added, the expression of a saliency region and saliency features is enhanced, and the detection accuracy of visual saliency detection is improved.

Description

technical field [0001] The invention relates to a visual saliency detection method based on deep learning, in particular to an attention mechanism-based asymmetric multimodal fusion saliency detection method. Background technique [0002] When looking for objects of interest in images, humans can automatically capture semantic information between objects and their contexts, pay high attention to salient objects, and selectively suppress unimportant factors. This precise mechanism of visual attention has been explained in various biological logic models. The goal of saliency detection is to automatically detect the most informative and attractive parts of an image. In many image applications, such as image quality assessment, semantic segmentation, image recognition, etc., identifying salient objects can not only reduce the computational cost, but also improve the performance of saliency models. Early saliency detection methods used manual features, that is, mainly for imag...

AI Technical Summary

Problems solved by technology

Most of the existing saliency detection methods have adopted the method of deep learning, using the method of combining the convolution layer and the pooling layer to extract image features, but the image features obtained by simply using the convolution operation and the pooling operation are not representative. characteristics, especially the pooling operation will lose the feature information of the image, which will lead to poor effect of the obtained saliency prediction map and low prediction accuracy

Images