Video generation model based on text description information and generative adversarial network

Abstract

The invention relates to a video generation model based on text description information and a generative adversarial network. According to the model, videos with the text description information are used as training data; a self-help sampling method is adopted to extract part of the videos in the training data and the text description information corresponding to the videos, and the videos and thecorresponding text description information are jointly input into an action recognition model for training; the remaining training data is input into the action recognition model for training after the corresponding text description information is removed, so that effective word vectors of the text description information in the training data are trained; and the word vectors and the videos are input into a provided generative adversarial network model, and the word vectors are used as limitation conditions so that a generator in the model can generate videos.

Description

technical field [0001] The present invention relates to the technical field of computer vision, and more specifically, relates to a video generation model based on text description information and a generation confrontation network. Background technique [0002] Image generation and video generation have always been a very important part of computer vision. In recent years, the technology of using generative models in machine learning to realize image generation has attracted much attention. Since Ian Goodfellow proposed Generative Adversarial Network (GAN) in 2014, After providing new ideas and methods for generative learning, both image generation and video generation technologies have been greatly improved. [0003] The present invention mainly involves the design of two parts, which are respectively the processing of text information and the design of generating confrontation network. [0004] The first is the processing of text information to obtain generalized and eff...

AI Technical Summary

Problems solved by technology

In 2016, M.Saito et al. began to apply GAN to video generation in the paper Temporal generative adversarial nets and C.Vondrick et al. Mapped to the same space, it can only handle videos of the same length, which is relatively inflexible

[0010] The existing GAN technology is mainly applied to image generation. Although only the dimension of time is added when generating video, it makes GAN very difficult to train. This is because video is a record of time and space. , the model not only needs to learn the appearance of the object as in image generation, but also needs to learn the laws of its motion to generate realistic videos

In addition, adding the dimension of time will bring huge changes. For example, the same person doing the same action at different speeds will be judged as a different video. The training mechanism proposed by MoCoGAN solves this problem to a certain extent, but The model only implements several simple videos about human activities and lacks generalization

Images