Visual question and answer method based on a combined relation attention network

Abstract

The invention discloses a visual question and answer method based on a combined relation attention network, aims at the problem that the existing visual question and answer method can only extract a simple visual relation, and innovatively constructs a self-adaptive relation attention module for fully extracting an accurate binary relation and a more complex ternary relation. The visual relationship between the relationship and the question can reveal deeper semantics, and the reasoning capability of the method when the method answers the question is enhanced. Meanwhile, the problem that an existing visual question and answer method cannot well fuse image features and position (relation) features of a target in an image is solved. According to the method, firstly, image features and position (relation) features of a target are extracted respectively, extraction of the image features of the target is independent of extraction of the relation features of the target, and then the two features are fused under guidance of a question, so that the two features are well fused together. By fully and accurately extracting the visual relationship and well fusing the image features and the relationship features, the accuracy of predicting the answers of the questions is improved.

Description

technical field [0001] The invention belongs to the technical field of visual question answering (VQA for short), and more specifically relates to a visual question answering method based on a combined relational attention network. Background technique [0002] In the existing technology, visual question answering (VQA) is mainly divided into two steps: 1) understand the content of image and text questions, extract image features and question features; 2) fuse image features and question features to obtain multimodal feature representation , and then predict the answer to the question through a softmax classifier. Among them, the attention mechanism (Attention) achieves the purpose of better understanding the image and the content of the question by focusing on the image area related to the question and the keywords in the question. [0003] In terms of feature fusion, at present, it is mostly based on bilinear network (Bilinear Network), which can well combine image featur...

AI Technical Summary

Problems solved by technology

[0006] 1. Existing visual question answering methods can only capture simple visual relationships, which are insufficient to answer complex textual questions;

[0007] 2. The existing visual question answering method uses the visual relationship features of the target to update the image features of the target. This method cannot well integrate the image features and relational features of the target in the image.

[0008] The above deficiencies result in inaccurate answers to prediction questions

Images