A combined video description method based on multi-modal features and multi-layer attention mechanism

Abstract

The invention discloses a combined video description method based on multi-modal features and multi-layer attention mechanism. Firstly, the invention counts the words appearing in the description sentence to form a vocabulary, and numbers each word to facilitate vector representation. Then three kinds of feature data are extracted, including semantic attribute feature, Image information features extracted by 2D-CNN and video motion information features extracted by 3D-CNN, and then multi-modal data dynamic fusion through the multi-layer attention mechanism to obtain visual information, and then according to the current context, adjust the use of visual information; Finally, according to the current context and visual information, the words described in the video are generated. After the multi-modal features of the video are fused through the multi-layer attention mechanism, the invention generates the semantic description of the video based on the multi-modal features of the video, thereby effectively improving the accuracy of the video description.

Description

technical field [0001] The invention belongs to the field of video description, and in particular relates to a combined video description method based on multi-modal features combined with a multi-layer attention mechanism. Background technique [0002] At present, the schemes used to generate description sentences for videos are mainly divided into template-based language methods and sequence learning methods. [0003] Among them, template-based language methods first align each sentence segment (e.g., subject, verb, object) with detected words from visual content, and then use predefined language templates to generate sentences that are highly dependent on the sentence template. The sentence pattern generated by this method is single, and sentences other than the language template cannot be generated. [0004] The sequence learning method is to design an encoding-decoding network, first use CNN (convolutional neural network) to encode the video, and then decode it through...

AI Technical Summary

Problems solved by technology

At present, the attention mechanism has been widely used in sequence learning methods. However, the current use of the attention mechanism is only limited to the single-modal feature after the feature extraction of the video, while ignoring the multi-modal features of the video itself. Therefore, it also makes the attention mechanism not be used effectively

Images