Method for solving video question-answer problems by using multi-granularity convolutional network self-attention context network mechanism

Abstract

The invention discloses a method for solving video question-answer problems by using a multi-granularity convolutional network self-attention context network mechanism. The method mainly comprises thefollowing steps that: 1) for a group of videos, frame-level and segment-level video expressions are obtained by using a pre-trained VGG network and a 3D-Conv network; 2) for question word embedding and answer word embedding of dialogue histories and new questions, the multi-granularity convolution self-attention network mechanism and a sentence-level context attention mechanism are adopted to obtain joint expressions related to the questions; and 3) a question-level time attention mechanism and a fused attention network mechanism are adopted to obtain joint video expressions related to the questions and generate answers to the questions asked about the videos. Compared with a common video question-answer solution, the method is advantageous in that the multi-granularity convolutional self-attention network is utilized, so that visible information and dialogue historical information can be combined to generate answers which better meet requirements. The method achieves a better effectin the video question-answer problems compared with a traditional method.

Description

technical field [0001] The invention relates to the generation of answers to video questions and answers, in particular to a method for solving video questions and answers by using a multi-granularity convolutional network self-attention context network mechanism. Background technique [0002] Video question answering is an important problem in the field of video information retrieval. The goal of this question is to automatically generate answers for related videos and corresponding questions. [0003] Existing technologies are mainly aimed at answering questions related to static image generation, and have achieved good results, but there are still great challenges in video question answering, such as contextual correlation between visible information and text information in video, And static images don't reflect this, ignoring a lot of contextual information. The present invention uses a self-attention mechanism to capture context information. Compared with the current R...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to solve the problems in the prior art, in order to overcome the lack of modeling of contextual information in the video in the prior art, and for the video often contains the appearance of the object and its movement information, and with the problem The relevant video information is scattered in some target frames of the video. The present invention provides a method for generating answers to video-related questions using a multi-granularity convolutional self-attention context network.

Questions and answers in video dialogue often contain contextual information, the present invention uses multi-granularity convolutional network self-attention context network to obtain context-aware question joint video expression

Images