Method for performing multi-mode video question answering by using frame-subtitle self-supervision

Abstract

The invention belongs to the field of video questions and answers, and particularly relates to a method for performing multi-mode video question answering by using frame-subtitle self-supervision. The method includes the following steps: extracting video frame features, question and answer features, subtitle features and subtitle suggestion features; obtaining frame features with attention and caption features with attention, and obtaining fusion features; calculating and obtaining a time attention score based on the fusion feature; calculating and obtaining the time boundary of the question by using the time attention score; calculating and obtaining answers to the questions by adopting the fusion features and the time attention scores; training a neural network by using the time boundary of the question and the answer of the question; and optimizing network parameters of the neural network, performing video question answering by using the optimal neural network, and delimiting a time boundary. The time boundary related to the problem is generated according to the self-designed time attention score instead of using time annotation with high annotation cost. In addition, more accurate answers are obtained by mining the relation between the subtitles and the corresponding video content.

Description

technical field [0001] The invention belongs to the field of video question answering, and in particular relates to a method for multimodal video question answering using frame-subtitle self-supervision. Background technique [0002] The multimodal video question answering task is a challenging task that currently attracts much attention. This task is designed in two fields of computer vision and natural language processing. It is required that the system can give the answer to the question for a specific video and delineate the corresponding time boundary of the question in the video. At present, the video question answering task is still a relatively novel task, and the research on it is still immature. [0003] Existing multi-modal video question answering tasks generally use convolutional neural network to encode video, and use recurrent neural network to encode question and answer and subtitles in the video. The question and answer encoding, subtitle encoding and video...

AI Technical Summary

Problems solved by technology

[0004] This scheme requires time tags to train the decoder to improve the effect, but the annotation of time tags is empirical and expensive

In addition, the above method separates video frames and subtitles, ignoring the correspondence between frames and subtitles

Images