Method of using multi-layer attention network mechanism to solve video question answering

Abstract

The invention discloses a method of utilizing a multi-layer attention network mechanism to solve video question answering. The method mainly includes the following steps: 1) for a group of videos, utilizing a pre-trained convolutional neural network to obtain frame-level and segment-level video expressions; 2) using a question-word-level attention network mechanism to obtain frame-level and segment-level video expressions for a question word level; 3) using a question-level time attention mechanism to obtain frame-level and segment-level video expressions related to a question; 4) utilizing aquestion-level fusion attention network mechanism to obtain joint video expressions related to the question; and 5) utilizing the obtained joint video expressions to acquire answers to the question asked for the videos. Compared with general video question answering solution, the method utilizes the multi-layer attention mechanism, and can more accurately reflect video and question characteristics, and generate the more conforming answers. Compared with traditional methods, the method achieves a better effect in video question answering.

Description

Technical field [0001] The present invention relates to video question and answer answer generation, and more particularly to a method for generating answers to video-related questions using a multi-layer attention network. Background technique [0002] Video question and answer questions are an important question in the field of video information retrieval. The goal of the question is to automatically generate answers for related videos and corresponding questions. [0003] The existing technology mainly solves question and answer questions related to static images. Although the current technology is aimed at static image question answering and can achieve good performance results, such a method lacks the modeling of time dynamic information in the video, so it cannot be well extended to video question answering tasks. [0004] In view of the situation that the video often contains the shape and movement information of the object, and the video information related to the problem is...

AI Technical Summary

Problems solved by technology

[0006] The purpose of the present invention is to solve the problems in the prior art, in order to overcome the lack of modeling of temporal dynamic information in the video in the prior art, and for the video often contains the shape of the object and its movement information, and The video information related to the question is scattered in some target frames of the video, the present invention provides a method of using a multi-layer attention network to generate an answer to the video-related question

Images