An image content understanding and visual question answering vqa method, storage medium and terminal

Abstract

The invention discloses an image content understanding and visual question answering VQA method, a storage medium and a terminal. The method includes the following steps: inputting images and questions to be answered into a trained prediction module for answering; the prediction module includes sequentially connected An attention module, a bilinear model and a classifier are fused which output the answer. The present invention follows the principles of "representing images and questions, performing feature representation on images and declarative sentences, fusing feature matrices, learning image features according to question features, learning image features according to correct declarative sentences, using correct declarative sentences to correctly guide the model , get the result" idea, complete the problem solving and visual question answering (VQA) tasks of image content; therefore, a fusion attention method of intensive interaction between image and question keywords is provided, which can learn the intensive interaction between image and text , so as to infer the relationship between the image and the question keyword.

Description

technical field [0001] The invention relates to the field of computer technology, and relates to an image content understanding and visual question answering VQA method, a storage medium and a terminal. Background technique [0002] Image content understanding and visual question answering (VQA) have attracted increasing interest in recent years. Multimodal fusion of global features is the most straightforward VQA solution. The general processing idea is to first represent the image and the question as a global feature, and then use the multimodal fusion model to predict the probability of the answer. [0003] In addition to understanding the visual content of images, VQA requires a full understanding of the semantics of natural language questions. Therefore, it is necessary to learn both textual attention to questions and visual attention to images. At present, the representation of the problem mainly uses LSTM, and the multi-modal fusion mainly uses the residual network...

AI Technical Summary

Problems solved by technology

[0004] Since the current network structure to solve the VQA problem is to learn its attention distribution in each modality separately, and then fuse it, there are several defects: (1) the network can only learn the rough interaction between multiple modalities, While ignoring the intensive interaction between images and text, the current collaborative attention is not enough to infer the relationship between images and question keywords; (2) the accuracy of the task of image question answering (VQA) is not high

Images