A method for multimodal video question answering using frame-subtitle self-supervision

Abstract

The invention belongs to the field of video question and answer, and in particular relates to a method for multi-modal video question and answer using frame-subtitle self-supervision. It includes the following steps: extracting video frame features, question and answer features, subtitle features, and subtitle suggestion features; obtaining frame features with attention and subtitle features with attention, and obtaining fusion features; calculating temporal attention scores based on fusion features; using temporal attention The time boundary of the question is calculated by the score calculation; the answer to the question is calculated by using the fusion feature and time attention score; the neural network is trained by using the time boundary of the question and the question answer; the network parameters of the neural network are optimized, and the optimal neural network is used for video question and answer and planning set time boundaries. Instead of using expensive time annotations, the present invention generates problem-related time boundaries based on self-designed time attention scores. In addition, the present invention obtains a more accurate answer by mining the relationship between the subtitle and the corresponding video content.

Description

technical field [0001] The invention belongs to the field of video question and answer, and particularly relates to a method for multi-modal video question and answer using frame-subtitle self-supervision. Background technique [0002] The multimodal video question answering task is a challenging task that currently attracts a lot of attention. This task is designed in the two fields of computer vision and natural language processing, and requires the system to give the answer to the question for a specific video and delineate the corresponding time boundary of the question in the video. At present, the video question answering task is still a relatively novel task, and the research on it is immature. [0003] At present, the existing multi-modal video question answering tasks generally use convolutional neural network to encode video, and use recurrent neural network to encode question and answer and subtitles in the video. Design the decoder, train the decoder with the q...

AI Technical Summary

Problems solved by technology

[0004] This scheme requires time tags to train the decoder to improve the effect, but the annotation of time tags is empirical and expensive

In addition, the above method separates video frames and subtitles, ignoring the correspondence between frames and subtitles

Images