Method for solving video questions and answers by use of hierarchical space-time attention coder-decoder network mechanism

Abstract

The invention discloses a method for solving video questions and answers by use of a hierarchical space-time attention coder-decoder network mechanism. The method mainly comprises the steps that (1) a space-time attention coding neural network is trained according to a video, question and answer training set, and coexpression of videos and questions is learned; and (2) a decoding neural network is trained according to output of the neural network which finishes coding and obtains video question coexpression together with relevant answers, and the decoding neural network is used for outputting corresponding natural language answers according to coexpression of the videos and the questions. Compared with ordinary video question and answer solutions, a time attention mechanism is utilized, a sequential relation between video frames is better utilized, meanwhile, a space attention mechanism is utilized to make key locations in the video frames precise, the characteristics of the videos and the questions can be reflected more accurately, and the answers better meeting requirements are generated. The effect of the method in terms of video question and answer problems is better than that of a traditional method.

Description

technical field [0001] The present invention relates to video question answering text generation, and more particularly to a method for generating answers to video-related questions using a layered spatio-temporal attention codec network. Background technique [0002] Open-ended video question answering is an important problem in the field of video information retrieval. The goal of this question is to automatically generate answers for related videos and corresponding questions. [0003] The existing technology mainly solves the question and answer questions related to static images. For the question and answer questions related to video, the method adopted is to simply shrink the video into an image question and answer question in time, and use the image question and answer method to solve the video question. Q&A questions. Although it can achieve good performance results for static image question answering, such a method cannot make good use of the sequence relationship ...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to solve the problems in the prior art. In order to overcome the deficiency that the mutual sequence relationship between the frames in the video cannot be well utilized in the prior art, the present invention provides a coding and decoding method using layered spatiotemporal attention. A method to generate relevant answers to open-ended video questions using a network of detectors

[0006] A layered spatio-temporal attention codec network is used to solve open-ended video question answering problems, including the following steps:

Images