Video description method based on multi-feature fusion

Abstract

The invention discloses a video description method based on multi-feature fusion. The method is characterized in that (1) deep spatial-temporal features of a video are extracted by fusing traditional CNN features and SIFT flow features; (2) an S2VT sentence generation model with added average pooling features being video overall features is adopted to generate corresponding sentence description according to the deep spatial-temporal features extracted in the step (1); and (3) a word2vec word vector method is adopted to replace one-hot vector word representation to optimize the sentence generation model in the step (2). The method has the advantages that more robust spatial-temporal features can be better extracted through multi-feature fusion; meanwhile, the average pooling features are added into the sentence generation model, so that more relations are established between visual information and words; finally, the word2vec word vector method is adopted to replace one-hot vector word representation, so that more relations are established between words, and video description performance is effectively improved.

Description

technical field [0001] The invention relates to video description technology, in particular to a video description method based on multi-feature fusion. Background technique [0002] Concept-Basic concept: Video description refers to describing the semantic information in the video with a natural language sentence based on the visual information of the given video. [0003] Purpose and meaning: The purpose of video description is to learn the semantic information contained in the video from the video information and describe it in natural language. It has wide application value in many fields, such as video retrieval and video annotation based on semantic content, descriptive video service, blind navigation and automatic video surveillance, etc. In recent years, with the rapid development of technologies such as the Internet and multimedia, the amount of visual data has grown exponentially, and the technology of learning semantic information from visual information has grad...

AI Technical Summary

Problems solved by technology

[0011] The purpose of the present invention is to propose a multi-feature fusion video description method in order to solve the problems existing in the existing video description method, which can better extract more robust spatio-temporal features, and at the same time add Overall features, in order to establish more connections between visual information and words, and finally use the word2vec word vector method to replace the one-hot vector word representation, to establish more connections between words, so as to better improve video description performance

Images