Visual question and answer method based on GAT relation reasoning

Abstract

The invention discloses a visual question and answer method based on GAT relation reasoning, and the method specifically comprises the steps: firstly, dividing a question into words for vectorization representation, and carrying out the sentence feature extraction, and obtaining a question feature vector; then, the Faster R-CNN is used in combination with the ResNet-101 network model to obtain object space coordinates and object categories, and the BUTD model obtains lt by using the object space coordinates and the object categories; an attribute class and an object class gt; the method comprises the following steps: obtaining a two-tuple, obtaining an edge label between objects by using a relation decoder, dynamically updating graph node information by using a question-guided graph attention convolutional network, and finally performing multi-modal fusion on graph representation and question features and inputting the fused graph representation and question features into a multi-layer perceptron to obtain answers. Ablation experiment verification is carried out on the GAT2R model on a data set, and compared with a reference model BUTD, the accuracy is improved.

Description

technical field [0001] The invention belongs to the technical field of image processing, and in particular relates to a visual question answering method based on GAT relational reasoning. Background technique [0002] The goal of Visual Question Answering (VQA) system is to answer questions based on the information provided by pictures. It has important research significance because of its rich application fields. Since the existing visual question answering methods focus on building new attention mechanisms and This makes the model more and more complex, ignoring research on problems that require spatial reasoning, semantic relations, and even scene understanding. Most VQA system frameworks mainly include image encoder, question encoder, multimodal fusion and answer prediction modules. Image representations are learned using convolutional neural networks and text representations are learned using recurrent neural networks, and then the two representations are fused into th...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to provide a visual question answering method based on GAT relational reasoning, which overcomes the problems that existing visual question answering methods ignore spatial reasoning, semantic relations and scene understanding

Images