Video question-answering method based on progressive graph attention network

An attention and network technology, applied in the field of video question answering, can solve problems such as low accuracy

An attention and network technology, applied in the field of video question answering, can solve problems such as low accuracy

CN112488055AActive Publication Date: 2021-03-12GUIZHOU UNIV +1
7 Cites 4 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Video question-answering method based on progressive graph attention network
  • Video question-answering method based on progressive graph attention network
  • Video question-answering method based on progressive graph attention network

Examples

Experimental program
Comparison scheme
Effect test

example

[0164] After experiments, it was found that the two multi-choice (Multi-Choice) sub-datasets in the existing large-scale video question answering dataset TGIF-QA have serious answer bias. These biases can have a great impact on the accuracy of the model. In order to solve this problem, this example builds a new dataset TGIF-QA-R on the basis of TGIF-QA. In this data set, candidate answers are independent of each other, and in this way, the impact caused by answer bias can be effectively reduced.

[0165] Test the effect of this method on three large benchmark data sets TGIF-QA, MSVD-QA and MSRVTT-QA and the newly constructed TGIF-QA-R data set, as can be seen from the experimental effect, the method proposed by the present invention is superior method at the highest level.

[0166] 1. Test results on TGIF-QA and TGIF-QA-R datasets

[0167]

[0168]

[0169] Table 1

[0170] It can be concluded from Table 1 that the present invention has achieved the best performance ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a video question-answering method based on a progressive graph attention network, and the method comprises the steps: sampling a novel progressive graph attention network, andexploring various types of visual interaction at a target level, a video frame level and a video segment level in a progressive mode; in the progressive graph attention network, the graph structure ofthe target level is mainly used for obtaining the space-time relationship between targets in the same frame or different frames, the graph structure of the video frame level explores the mutual relationship between video frames, and the graph structure of the video clip level constructs the time sequence relationship between different actions. Meanwhile, an attention mechanism is also used for paying attention to graph vertexes and edges related to problems, and graph features of different levels are connected in a progressive mode. In this way, each graph can focus on its spatio-temporal contiguous vertices and finer grained visual content based on visual correlations. Therefore, the accuracy of predicting the answers to the questions is improved.

Description

technical field [0001] The invention belongs to the technical field of Video Question Answering (Video-QA), and more specifically relates to a video question answering method based on a progressive graph attention network. Background technique [0002] In the prior art, the main goal of Video Question Answering (Video-QA) is to answer natural language questions related to video content. Therefore, the understanding of video content is very important. The classic video question answering method is mainly divided into three steps: 1) using convolutional neural network (CNN) model and recurrent neural network (RNN) model to extract video features and question features respectively; 2) under the guidance of question features, Focus on the part of the video features that is relevant to answering questions, so as to obtain a more expressive video representation; 3) Fusion of video features and question features to obtain a multi-modal feature representation, and then predict the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
12 Mar 2021
Publication
CN112488055A
IPC
G06K9/00; G06K9/46; G06K9/62; G06N3/04; G06N3/08
CPC
G06N3/049; G06N3/08; G06V20/41; G06V10/44; G06N3/045; G06F18/2415; G06F18/253
Inventors
杨阳; 彭亮