Video description method based on multi-feature fusion

Abstract

The invention discloses a video description method based on multi-feature fusion, which is characterized in that: 1) extracting deep spatiotemporal features of video by fusing traditional CNN features and SIFT stream features; 2) extracting deep spatiotemporal features according to step 1), using Add the S2VT sentence generation model with the average pooling feature as the overall feature of the video to generate the corresponding sentence description; 3) use the word2vec word vector to replace the sentence generation model in step 2) of the one-hot vector word representation optimization. The advantage of this method is that through multi-feature fusion, more robust spatio-temporal features can be better extracted. At the same time, average pooling features are added to the sentence generation model to establish more connections between visual information and words. Finally, word2vec word vectors are used. The method replaces the one-hot vector word representation, establishes more connections between words, and effectively improves the performance of video description.

Description

technical field [0001] The invention relates to video description technology, in particular to a video description method based on multi-feature fusion. Background technique [0002] Concept-Basic concept: Video description refers to describing the semantic information in the video with a natural language sentence based on the visual information of the given video. [0003] Purpose and meaning: The purpose of video description is to learn the semantic information contained in the video from the video information and describe it in natural language. It has wide application value in many fields, such as video retrieval and video annotation based on semantic content, descriptive video service, blind navigation and automatic video surveillance, etc. In recent years, with the rapid development of technologies such as the Internet and multimedia, the amount of visual data has grown exponentially, and the technology of learning semantic information from visual information has grad...

AI Technical Summary

Problems solved by technology

[0011] The purpose of the present invention is to propose a multi-feature fusion video description method in order to solve the problems existing in the existing video description method, which can better extract more robust spatio-temporal features, and at the same time add Overall features, in order to establish more connections between visual information and words, and finally use the word2vec word vector method to replace the one-hot vector word representation, to establish more connections between words, so as to better improve video description performance

Images