Image question-answering method based on multi-scale deep learning

Abstract

The invention discloses an image question-answering method based on multi-scale deep learning. The method is enlightened by the cognitive behaviors of people during the image question-answering process. The method copmrises the following steps of 1) setting the same picture as three pictures different in scale size according to requirements, adopting a pre-trained convolution neural network for extracting picture features and obtaining multi-scale picture features; 2) obtaining the feature expression of an interrogative sentence by adopting a recursion neural network, and acquiring n-element interrogative sentence feature expression by connecting a convolution layer with different convolution kernel sizes; 3) exploring the built-in incidence relation between picture features of different scales and n-element interrogative sentence feature expression, namely the similarity measurement; 4) finally, fusing picture features of different scales and n-element interrogative sentence features,and deducing the answer of a predicted problem from the large scale to the middle scale and the small scale by adopting a hierarchical network structure. According to the invention, the cognitive behaviors of people during the image question-answering process are simulated. Meanwhile, the high precision is obtained on the basis of a reference data set.

Description

technical field [0001] The invention relates to the multi-modal technical field of image processing and natural language processing, in particular to an image question answering method based on multi-scale deep learning. Background technique [0002] Image question answering is a challenging task that has emerged in recent years. Given a picture and a question related to the picture, the image question answering task can automatically generate an answer. Image question answering has many potential applications, such as helping blind or visually impaired people access information from websites or the real world. In addition, it can be used for human-computer interaction to interrogate visual content, etc. [0003] At present, most of the algorithms that have been proposed to solve image question answering tasks use convolutional neural networks to extract the features of the entire image and recurrent neural networks to extract the features of question sentences, and then le...

AI Technical Summary

Problems solved by technology

However, in the current image question answering system, there is no model that attempts to explore the effectiveness of multi-scale image features in image question answering and explain how the deep neural network model for image question answering predicts the answer.

Images