Visual question and answer method optimized by using position information

Abstract

The invention discloses a visual question and answer method optimized by using position information. The method comprises the following steps: collecting training data including pictures and questions related to a given picture; carrying out pre-processing on the input problem; carrying out pre-processing on the input picture; performing multi-head position self-attention operation to obtain a fusion feature vector of words in the question: performing position self-attention operation, and fusing a visual mode and a language mode by using a position joint attention mechanism to obtain a fusion feature vector of an object in the picture; the fusion feature vector of the object and the fusion feature vector of the word are compressed and then fused; the method comprises the following steps: constructing a visual question-answering model, predicting answers to questions, calculating differences between the answers and true values, training the visual question-answering model through back propagation, and inputting data into the trained visual question-answering model to perform visual question-answering. The visual question answering method provided by the invention can better understand questions, and is helpful for the model to understand sentence semantics.

Description

technical field [0001] The invention relates to the technical field of visual question answering, in particular to a visual question answering method optimized by using position information. Background technique [0002] Visual question answering is a research hotspot in the field of artificial intelligence in recent years, which has attracted extensive attention of scholars. Given a picture and a question related to the content of the picture, the task requires the machine to correctly understand the image and the question, and make an answer to the question. Answer appropriately. This task requires the model to have a fine-grained understanding of visual and linguistic information, and to be able to perform cross-modal reasoning like a human (associate objects in pictures with words in questions, understand logical relationships in questions, and be able to combine The content in the figure infers the corresponding answer). The research results in this field have been wi...

AI Technical Summary

Problems solved by technology

In recent years, with the prosperity and development of the image and natural language communities, the cross-field of visual question answering is also full of various methods: 1) The multi-modal fusion method based on the attention mechanism can allow the model to focus on key objects in the picture and relevant information in the text. Nouns, but the model lacks a non-selection mechanism, and the normalized operation in the attention will lead to the inability to introduce quantitative information into the model

2) Based on the method of module network imitating the human reasoning process, this method performs grammatical analysis on the problem, designs several types of basic module networks, and inputs the problem into the corresponding module for processing according to the analyzed results, which has good interpretability. But the effect on the real picture data set is not good

3) Based on the method of graph learning, this method regards the objects in the picture as nodes, and the relationship between objects as edges to build a graph, but the effect is not much improved

4) The method of introducing external knowledge, there are many questions that require external knowledge to answer, but how to make the model better learn external knowledge is still difficult

5) For the optimization of the counting problem of the attention mechanism, etc., the problem that the aforementioned attention mechanism cannot introduce counting information into the model has been improved to a certain extent, but the scope of application is small

However, this method uses a matching algorithm between the visual modal scene graph and the language modal text graph to predict the answer, which is similar to the module network that was popular in academia a few years ago, and has good interpretability, but due to the complexity of the real scene , the effect of this method is not as good as that of various end-to-end methods that are popular today, and this method only performs matching operations between visual modalities and language modalities, and lacks steps for reasoning about problems

Images