Double-flow video generation method based on different feature spaces of text

Abstract

The invention discloses a double-flow video generation method based on different feature spaces of a text. The double-flow video generation method comprises a text feature extraction process, a double-flow video generation process and an adversarial resistance training process. According to the method, the input text content is analyzed, the feature information obtained by using the attention mechanism is separated, and the information of the appearance feature space and the information of the motion feature space are extracted respectively; a double-flow video generation mode is adopted, so that the learning capability of a single model on specified features is utilized to the maximum extent. According to the method, the adversarial resistance training process is used, the model is guidedand trained from the two aspects of the quality of each frame and the coherence of overall motion, and meanwhile text features are added to strengthen the training process, so that the generated video can conform to input text description, and the generation accuracy is achieved. The method provided by the invention can realize intelligent generation, and has wide market demands and application prospects in the aspects of material accumulation and data set automatic generation.

Description

technical field [0001] The present invention relates to the technical fields of pattern recognition, natural language processing, computer vision, etc., and in particular to a video generation method based on different feature spaces of texts, intelligently generating A video clip that matches the text content. Background technique [0002] In recent years, with the rapid development of computer vision and the introduction of generative adversarial networks, the research on image generation has received more and more attention. It has very positive significance in material accumulation and automatic generation of data sets. Compared with images, videos are more vivid and difficult to generate, so the exploration of video generation is more meaningful. At the same time, if randomly generating videos like most image generation methods does not have much practical value, users will more likely want to generate based on some given information, for example, the user inputs "a pe...

AI Technical Summary

Problems solved by technology

[0003] Most of the existing methods uniformly process the entire text, and directly generate the entire video clip from the extracted features. However, such processing overestimates the learning ability of a single model, which requires not only learning spatial features (appearance information), but also Learning temporal features (motion information) makes the model unable to fully understand both features, and it is difficult to effectively generate high-quality

Images