Method for solving video question and answer tasks needing common knowledge by using question-knowledge guided progressive space-time attention network

Abstract

The invention discloses a method for solving a video question and answer task needing common knowledge by using a question-knowledge guided progressive space-time attention network, which comprises the following steps: for a video, obtaining a video object set by using a Faster-RCNN; retrieving an annotation text corresponding to the video object set in an external knowledge base to obtain external knowledge; extracting semantic features of external knowledge by using Doc2Vec to obtain a knowledge feature set of the video; aiming at the problem, converting an input word into a word embedding vector by using an embedding layer (embedding layer); inputting the word embedding vector into a progressive space-time attention network to generate an answer; by using the additional information, more specific questions, such as some common questions, can be answered; external knowledge and questions are combined, progressive video attention is guided in space and time dimensions, and fine-grained joint video representation is learned to perform answer prediction.

Description

technical field [0001] The invention relates to the field of video question answering answer generation, in particular to a method for solving video question answering tasks requiring common sense using a question-knowledge guided progressive spatiotemporal attention network. Background technique [0002] Visual Question Answering (VQA) is a task for bridging computer vision (CV) and natural language processing (NLP), which automatically returns accurate answers from reference visual content according to a user's question. According to the type of visual content, there are mainly two kinds of visual question answering, one is image question answering, and the other is video question answering. In recent years, a lot of work has been done in the field of visual question answering. However, most existing work focuses on static image question answering. [0003] Video question answering is a nascent field in which far less work has been done by researchers than image question...

AI Technical Summary

Problems solved by technology

However, this way is a rough representation of the visual content and lacks more detailed information such as objects in the frame

This would render this approach insufficient for answering questions that depend on the details of the video content

Images