Video question-answering method based on progressive graph attention network

Abstract

The invention discloses a video question-answering method based on a progressive graph attention network, and the method comprises the steps: sampling a novel progressive graph attention network, andexploring various types of visual interaction at a target level, a video frame level and a video segment level in a progressive mode; in the progressive graph attention network, the graph structure ofthe target level is mainly used for obtaining the space-time relationship between targets in the same frame or different frames, the graph structure of the video frame level explores the mutual relationship between video frames, and the graph structure of the video clip level constructs the time sequence relationship between different actions. Meanwhile, an attention mechanism is also used for paying attention to graph vertexes and edges related to problems, and graph features of different levels are connected in a progressive mode. In this way, each graph can focus on its spatio-temporal contiguous vertices and finer grained visual content based on visual correlations. Therefore, the accuracy of predicting the answers to the questions is improved.

Description

technical field [0001] The invention belongs to the technical field of Video Question Answering (Video-QA), and more specifically relates to a video question answering method based on a progressive graph attention network. Background technique [0002] In the prior art, the main goal of Video Question Answering (Video-QA) is to answer natural language questions related to video content. Therefore, the understanding of video content is very important. The classic video question answering method is mainly divided into three steps: 1) using convolutional neural network (CNN) model and recurrent neural network (RNN) model to extract video features and question features respectively; 2) under the guidance of question features, Focus on the part of the video features that is relevant to answering questions, so as to obtain a more expressive video representation; 3) Fusion of video features and question features to obtain a multi-modal feature representation, and then predict the ...

AI Technical Summary

Problems solved by technology

[0004] Most of the existing methods use attention mechanism (Attention) or graph network structure (GCN) to explore the single interaction between objects or frames in the video. However, these interactions are often not enough to represent the complex scenes in the video, because in In the video, not only the spatio-temporal relationship between the targets and the relationship between the video frames, but also the timing relationship of the actions in it, so the accuracy of the answer to the prediction question is low

Images