Dual-channel semantic positioning multi-granularity attention mutual enhancement video question answering method and system

Abstract

The invention discloses a dual-channel semantic positioning multi-granularity attention mutual enhancement video question answering method and system. The method comprises the following steps: constructing a feature extraction coding module to extract features; constructing a coarse and fine granularity global retrieval module to perform attention calculation on the features; constructing a graph attention network module to enhance key information between the features; constructing a local positioning module to position key feature information of different channels; and constructing a fusion attention module to perform multi-level fusion on the enhanced features, and finally obtaining answer index information through an answer prediction module. According to the method, feature information of different granularities in a video is defined as visual semantic and text semantic channels through multi-module design, an auxiliary positioning mechanism is designed for different channels, and feature information most related to a problem is obtained by enhancing sharing representation. Correlation mutual attention is carried out on global and local information by utilizing a graph attention network, so that characteristics related to questions under the same time and space are optimally expressed, and the difficulty of current video question answering is relieved.

Description

technical field [0001] The invention relates to the field of video question answering, in particular to a video question answering method and system for dual-channel semantic positioning, multi-granularity attention mutual enhancement. Background technique [0002] Video has become the most important multimedia information carrier at present and is widely used in people's lives. Due to the characteristics of time and space, video has the characteristics of high information density, diverse categories, changeable content, and complex structure. Users cannot quickly locate the results of the video content they are interested in, and at the same time, it brings new challenges to video understanding tasks such as video description, classification, and recognition. Video question answering (VideoQA) is a fine-grained video understanding task after video description. Compared with the general description in video description task, video question answering not only needs to be able...

AI Technical Summary

Problems solved by technology

The main problems to be solved by the present invention are as follows: firstly, the existing models use graphs to model video when they do not pay attention to the problem, resulting in excessive redundant information in the graphs, causing disturbances between relationships, and poor model performance The second is that the existing models lack fine-grained representation and contextual information connection; the third is that the existing models have insufficient problems in the multi-channel feature fusion and reasoning mechanism of video

Images