Method for solving video question and answer problem by utilizing graph theory-based multi-interaction network mechanism

Abstract

The invention discloses a method for solving a video question and answer problem by utilizing a graph theory-based multi-interaction network mechanism. The method comprises the following steps: 1) obtaining a frame-level video expression by utilizing a ResNet network for a video; 2) utilizing a MaskR-CNN network to obtain existence and position characteristics of the object; 3) extracting problemword-level information by using a GloVe network; 4) using a GNN network composition based on a graph theory, introducing a message mechanism to iterate the graph, and finally obtaining object existence and inter-object connection feature expression; 5) introducing multiple interactions, and learning by using a feedforward neural network to obtain feature expressions of object existence and inter-object dynamic relationship related to the problem and video frame-level and fragment-level expressions; and 6) using different strategies for different types of question and answer modules. Accordingto the method, the space-time dependence relation and the dynamic semantic interaction information between the objects are obtained through the mechanism, the deeper effect is achieved in video understanding, and then more accurate answers are given.

Description

technical field [0001] The invention relates to the generation of video question-and-answer answers, in particular to a method for solving video question-and-answer questions by using a graph theory-based multi-interaction network mechanism. Background technique [0002] Video question answering is an important problem in the field of video information retrieval. The goal of this question is to automatically generate answers for related videos and corresponding questions. [0003] Existing techniques mainly address question-answering questions related to static images. Although the current technology can achieve good performance results for static image question answering, such methods lack the modeling of temporal dynamic information in videos, so they cannot be well extended to video question answering tasks. [0004] For the situation that the video often contains the shape of the object and its movement information, and the video information related to the problem is sc...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to solve the problems in the prior art, in order to overcome the lack of modeling of temporal dynamic information in the video in the prior art, and for the video often contains the shape of the object and its movement information, and The video information related to the question is scattered in some target frames of the video. The present invention provides a method for solving video question-answering questions using a multiple interactive network mechanism based on graph theory. The specific technical solution adopted by the present invention is :

Images