Method for solving video question and answer by using hierarchical codec network mechanism

Abstract

The present invention discloses a method for solving open-type long video question and answer by using a hierarchical codec network mechanism, which mainly comprises the following steps: 1) training an adaptive hierarchical coding neural network for a group of video, question and answer training set, segmenting the long video based on a question and video learning adaptive segmentation mechanism,thereby obtaining a joint expression of the video segment and the question; 2) for the output of the coded neural network obtaining the joint expression of the video and the question, combining the idea of reinforcement learning together with a related answer to train a decoding neural network, directed at a natural language answer corresponding to the output of the joint expression of the video and the question. Compared with the general video question and answer solution, the method for solving open-type long video question and answer by using a hierarchical codec network mechanism in the present invention utilizes question-based adaptive layering, which can better lock the segments favorable for answering questions in long videos, can better reflect the characteristics of the video, andutilizes the reinforcement learning mechanism to train the decoder, so that a more powerful decoder can be obtained, and an answer better satisfying the requirement is produced. The effect achieved by the method in the present invention in the long video question and answer questions is better than the traditional method.

Description

technical field [0001] The present invention relates to video question and answer answer generation, and in particular to a method for generating answers to video-related questions using a layered codec network mechanism. Background technique [0002] Open-ended video question answering is an important problem in the field of video information retrieval. The goal of this question is to automatically generate answers for related videos and corresponding questions. Open-ended video question answering is the fundamental problem of visual question answering, which automatically generates natural language answers from quoted video content according to a given question. [0003] Most of the current video question answering methods mainly focus on short video question answering questions, and most of their methods learn the semantic representation of the video from the LSTM network layer, and then generate the answer. Although the current technology has achieved good results for s...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to solve the problems in the prior art. In order to overcome the lack of modeling of video semantic features in long video question and answer in the prior art, it is aimed at content with different semantics between multiple frames in long video, and these contents In the case of different segments scattered in the video, the present invention provides a method for solving open long video question answering using the adaptive layered reinforcement learning codec network mechanism

[0007] The layered codec network mechanism is used to solve the open long video question answering problem, which includes the following steps:

Images