A spatio-temporal and channel-based multi-attention mechanism video description method

Abstract

The invention discloses a multi-attention mechanism video description method based on space-time and channel, wherein the video features are extracted through a CNN network, the video features and theoutput of the encoded last time are calculated based on the multi-attention network, thus, the attention weights of the video features in time domain, space domain and channel are obtained, and thenthe three weights are calculated again with the video features to obtain the features of fusion, so that we can obtain more effective video features, and finally, the fusion features are coded and output to obtain a more consistent description with the video content.

Description

technical field [0001] The invention belongs to the technical field of optical communication, and more specifically relates to a multi-attention mechanism video description method based on time, space and channels. Background technique [0002] Video description is a research in two fields of computer vision and natural language processing, which has received great attention in recent years. Venugopalan released a video description model based on the "encoding-decoding" framework in 2014. The encoding model in the paper first uses CNN to extract features for a single video frame, and then adopts two encoding models of mean pooling and time sequence encoding. Although the model has been successfully applied to video description, there are still some problems in the video description model: [0003] The first problem is that video features are not effectively utilized. In the paper, the video features are only used in the first decoding, and the video features are not used ...

AI Technical Summary

Problems solved by technology

[0003] The first problem is that there is no effective use of video features

In the paper, the video features are only used in the first decoding, and the video features are not used in the subsequent moments, which leads to the weakening of the impact of video features on word prediction when the time sequence increases, which will reduce the ability of the model to generate sentences

[0004] A direct solution to this problem is to add video features every time, but since the video features are continuous multiple images, if the mean pooling method is still used to send the decoding model every moment, obviously this is still not effective Take advantage of video features

[0006] The second problem is the consistency of visual content features and sentence descriptions

The first problem is that although the method based on temporal attention improves the utilization of video features, but in a deeper way, this method still does not fully model the relationship between video features and sentence descriptions, which leads to The second problem that comes is how to ensure the consistency of visual content feature sentence description

Images