Visual question answering method and device based on question semantic mapping

Abstract

The invention discloses a visual question answering method and device based on question semantic mapping. The method comprises the steps of extracting visual features of images in a training set and question features in questions; carrying out feature fusion on the extracted visual features and problem features; classifying questions in the training set, and counting answer ranges of different question types in the training set; extracting answer features of answers in the answer range; establishing a mapping relation between the question features and the answer features, and obtaining answer range probability distribution; carrying out reasoning according to fusion features after feature fusion and answer range probability distribution, and obtaining a final answer. The method comprises the following steps: extracting question features through a gating loop unit, extracting consistent semantics of a question and an answer by utilizing embedded learning, mapping the semantics to the answer, and extracting high-level semantic information of an image through a question-oriented regional attention mechanism and a relationship; and fusing the image high-level semantic information with the answer range mapping result to generate a final answer.

Description

technical field [0001] The invention relates to the technical field of machine learning, in particular to a visual question answering method and device based on question semantic mapping. Background technique [0002] Visual question answering is a popular research direction in the field of computer vision. Visual question answering is to input an image and a question related to the image. The model needs to understand the image and question, and then output the answer to the question. Therefore, the same question has different answers in different images, and the same image has different understanding emphases in different question contexts. This means that visual question answering can understand the content of the image in different ways according to different question contexts, and make necessary inferences about the information that cannot be directly obtained in the image, such as the position and size relationship between the subjects involved, and get the final answe...

AI Technical Summary

Problems solved by technology

[0004] Most of the existing methods extract the relevant features in the image through the attention mechanism, etc., and fuse the information of two different modalities. However, the current methods still have some shortcomings.

Most methods ignore the fact that the question itself plays an important role in the derivation of the answer. The type and method of the question determine the range of the answer. The current methods often ignore the mapping relationship between the question and the answer range.

Images