Video generation model training method, video generation method and device
Through iterative training of video generation model, combining coding module, timing module and content guidance module, the problems of poor video quality and poor user experience in the existing technology are solved, efficient and low-cost video generation is achieved, and the content generation is improved. Consistency and detail characteristics.
Patent Information
- Application Number
- CN202411176244.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Existing technologies for generating videos based on text and/or pictures have problems such as poor video quality, poor user experience, and the generated content does not meet user expectations, making it difficult for users to accurately control the video generated content.
By obtaining training samples including continuous picture sequences and text descriptions, we determine reference information such as reference pictures and similarity information, iteratively train the video generation model, adopt the backbone network of the encoding module and the timing module, and combine it with the content guidance module to gradually improve the video generation model The resolution and detail characteristics of the model are adjusted to improve the quality of video generation.
It significantly improves the quality of video generation and user experience, enhances the consistency between the generated video and the input content, reduces the cost and threshold of video production, and realizes a more efficient video generation process.
Smart Images

Figure CN119052597A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a video generation model training method, a video generation method and a device. Background Art
[0002] Text-to-video refers to the technology that converts text descriptions into video content. Similarly, image-to-video refers to the technology that converts images into video content. This technology can provide creative inspiration and tool support for content creators. Creators can generate corresponding video content from text and images, and use it to make short videos, animations, or generate virtual anchors, etc.
[0003] Using generative AI to make videos is a highly efficient and low-cost technology that greatly reduces the threshold for video production and editing. However, the current technology for generating videos based on text and / or images still has many shortcomings. The quality of the generated videos is poor, the user experience is poor, and the generated video content often does not meet user expectations. It is difficult for users to accurately control the generated video content through input. Therefore, a technical solution is urgently needed to solve these problems existing in the existing technology. Summary of the invention
[0004] One or more embodiments of the present specification provide a method for training a video generation model, comprising: obtaining a training sample, wherein the training sample comprises a continuous picture sequence and a text description suitable for describing the picture sequence; determining reference information according to the picture sequence, wherein the reference information comprises a reference image related to the picture sequence and similarity information between the reference image and the picture sequence; iteratively training the video generation model according to the training sample and the reference information to obtain a video generation model of a first stage and a corresponding first model weight; using the first model weight as an initial weight, and iteratively training the video generation model obtained in the first stage according to the training sample and the reference information to obtain a video generation model of a second stage and a corresponding second model weight, wherein the resolution of the picture sequence and the reference image in the second stage is greater than the resolution of the picture sequence and the reference image in the first stage.
[0005] According to the method provided by one or more embodiments of this specification, the reference information also includes detail information extracted from the reference image.
[0006] According to the method provided by one or more embodiments of the present specification, determining reference information based on the picture sequence includes: determining any one picture in the picture sequence as a reference picture; extracting high-frequency details in the reference picture; and calculating the similarity coefficient between the reference picture and each picture in the picture sequence based on structural similarity.
[0007] The method provided according to one or more embodiments of the present specification also includes: preprocessing the picture sequence so that the image resolution of the picture sequence in the second stage is greater than the image resolution of the picture sequence in the first stage; and determining reference information of the first stage and the second stage respectively based on the preprocessed picture sequence.
[0008] According to the method provided by one or more embodiments of the present specification, the video generation model includes an encoding module and a backbone network including a timing module, the encoding module is suitable for encoding the training samples, the reference images and the detail information, the backbone network is suitable for predicting the noise distribution according to the input noisy image based on the diffusion principle, and the timing module is suitable for capturing the temporal correlation between images.
[0009] The method provided according to one or more embodiments of the present specification further includes: determining the similarity information according to the encoded reference image and the encoded picture sequence.
[0010] According to the method provided by one or more embodiments of the present specification, iterative training of the video generation model according to the training samples and the reference information includes: performing noise processing on the encoded picture sequence to obtain a noisy image and input noise; inputting the noisy image, the similarity information, the encoded reference image, the encoded detail information, the encoded text description and the preset timestamp into the backbone network; obtaining the output noise corresponding to the preset timestamp output by the backbone network; adjusting the weight of the first-stage video generation model according to the input noise and the output noise; and repeating the above process for iterative training.
[0011] According to the method provided by one or more embodiments of the present specification, adjusting the weight of the video generation model according to the input noise and the output noise includes: determining loss parameter information according to the input noise and the output noise; and adjusting the weight of the video generation model according to the loss parameter information.
[0012] The method provided according to one or more embodiments of the present specification also includes: combining the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network; adjusting the weights of the unfrozen input layer in the backbone network and the weights of the timing module according to the loss parameter information.
[0013] According to the method provided by one or more embodiments of the present specification, the video generation model also includes a content guidance module, and the method also includes: determining content guidance information through the content guidance module, wherein the content guidance information is determined based on a reference image and a text description; using the first model weight as the initial weight, and iteratively training the video generation model obtained in the first stage according to the training samples, reference information and the content guidance information to obtain the video generation model of the second stage and the corresponding second model weight.
[0014] According to the method provided by one or more embodiments of the present specification, determining content guidance information through a content guidance module includes: extracting image feature information in a reference image through the content guidance module, wherein the image feature information is suitable for expressing the content presented in the reference image; mapping the image feature information to the same dimension as the encoded text description; and splicing the image feature information of the same dimension with the text description to obtain content guidance information.
[0015] According to the method provided by one or more embodiments of the present specification, iterative training of the video generation model obtained in the first stage according to the training samples, the reference information and the content guidance information includes: performing noise processing on the encoded picture sequence to obtain a noisy image and input noise; inputting the noisy image, the similarity information, the encoded reference image, the encoded detail information, the content guidance information and the preset timestamp into the backbone network; obtaining the output noise corresponding to the preset timestamp output by the backbone network; adjusting the weight of the second stage video generation model according to the input noise and the output noise; and repeating the above process for iterative training.
[0016] One or more embodiments of the present specification also provide a video generation method, including: obtaining input content, the input content including an input reference image and a text description corresponding to the reference image; determining reference information based on the reference image, the reference information including similarity information related to the input content; inputting the input content and the reference information into a pre-trained first-stage video generation model to obtain a first generated video; obtaining a selected first generated video; inputting the input content, the reference information and the selected first generated video into a pre-trained second-stage video generation model to obtain a second generated video, wherein the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
[0017] According to the method provided by one or more embodiments of this specification, the reference information also includes detail information extracted from the reference image.
[0018] According to the method provided by one or more embodiments of the present specification, determining reference information according to the input content includes: extracting high-frequency details in the reference image; and generating a similarity coefficient related to the input content through a preset periodic function.
[0019] The method provided according to one or more embodiments of the present specification also includes: preprocessing the reference image so that the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model; and determining reference information input into the first-stage video generation model and the second-stage video generation model respectively based on the preprocessed reference image.
[0020] According to the method provided by one or more embodiments of the present specification, before obtaining the selected first generated video, it also includes: replacing the random seed and generating more than two first generated videos.
[0021] According to the method provided by one or more embodiments of the present specification, the video generation model includes an encoding module and a backbone network including a timing module, the encoding module is suitable for encoding the input content and the detail information, and decoding the image features output by the backbone network, the backbone network is suitable for iteratively denoising the input noisy image based on the diffusion principle and outputting image features, and the timing module is suitable for capturing the temporal correlation between images.
[0022] According to the method provided by one or more embodiments of the present specification, the input content and the reference information are input into a pre-trained first-stage video generation model to obtain a first generated video, including: obtaining two or more preset noises consistent with the dimensions of the encoded reference image; inputting the two or more preset noises, the encoded reference image, the encoded detail information, the encoded text description, the similarity information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a first image sequence latent vector; decoding the first image sequence latent vector through the encoding module to obtain a first generated video.
[0023] According to the method provided by one or more embodiments of the present specification, the second-stage video generation model also includes a content guidance module, and the method also includes: determining content guidance information through the content guidance module, wherein the content guidance information is determined based on a reference image and a text description; inputting the input content, the reference information, the content guidance information and the selected first generated video into the backbone network for iterative noise reduction processing to obtain a second generated video.
[0024] According to the method provided by one or more embodiments of the present specification, determining content guidance information through a content guidance module includes: extracting image feature information in a reference image through the content guidance module, wherein the image feature information is suitable for expressing the content presented in the reference image; mapping the image feature information to the same dimension as the encoded text description; and splicing the image feature information of the same dimension with the text description to obtain content guidance information.
[0025] According to the method provided by one or more embodiments of the present specification, the input content, the reference information, the content guide information and the selected first generated video are input into the backbone network for iterative noise reduction processing to obtain the second generated video, including: adding noise based on the selected first generated video to obtain a noisy image sequence; inputting the noisy image sequence, the encoded reference image, the encoded detail information, the similarity information, the content guide information and a preset timestamp into the backbone network for iterative noise reduction processing to obtain a second image feature sequence; decoding the second image feature sequence through the encoding module to obtain the second generated video.
[0026] One or more embodiments of the present specification also provide a training device for a video generation model, comprising: an acquisition module, used to acquire training samples, wherein the training samples include a continuous picture sequence and a text description suitable for describing the picture sequence; a reference information determination module, used to determine reference information based on the picture sequence, wherein the reference information includes a reference image related to the picture sequence and similarity information between the reference image and the picture sequence; a first-stage training module, used to iteratively train the video generation model according to the training samples and the reference information to obtain a first-stage video generation model and a corresponding first model weight; a second-stage training module, used to use the first model weight as the initial weight, and iteratively train the video generation model obtained in the first stage according to the training samples and the reference information to obtain a second-stage video generation model and a corresponding second model weight, wherein the resolution of the picture sequence and the reference image in the second stage is greater than the resolution of the picture sequence and the reference image in the first stage.
[0027] One or more embodiments of the present specification also provide a video generation device, including: a content acquisition module, used to acquire input content, wherein the input content includes an input reference image and a text description corresponding to the reference image; a reference information determination module, used to determine reference information based on the reference image, wherein the reference information includes similarity information related to the input content; a first video generation module, used to input the input content and the reference information into a pre-trained first-stage video generation model to obtain a first generated video; a video acquisition module, used to acquire a selected first generated video; a second video generation module, used to input the input content, the reference information and the selected first generated video into a pre-trained second-stage video generation model to obtain a second generated video, wherein the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
[0028] One or more embodiments of the present specification also provide an electronic device, wherein the computer device includes a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the training method of the video generation model described in some embodiments of the present specification, or to implement the video generation method described in some embodiments of the present specification.
[0029] One or more embodiments of the present specification also provide a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the training method of the video generation model described in some embodiments of the present specification, or to implement the video generation method described in some embodiments of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] This specification will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. The same numbers in the drawings represent the same structures or steps.
[0031] Figure 1 It is a schematic diagram of an application scenario of a video generation model according to some embodiments of this specification.
[0032] Figure 2 This is an exemplary flowchart of a video generation model training method shown in some embodiments of this specification.
[0033] Figure 2a It is a schematic diagram of the first-stage training method of the video generation model shown in some embodiments of this specification.
[0034] Figure 2bIt is a schematic diagram of the second-stage training method of the video generation model shown in some embodiments of this specification.
[0035] Figure 3 This is an exemplary flowchart of another video generation model training method shown in some embodiments of this specification.
[0036] Figure 4 This is an exemplary flowchart of another video generation model training method shown in some embodiments of this specification.
[0037] Figure 5 This is an exemplary flowchart of another video generation model training method shown in some embodiments of this specification.
[0038] Figure 6 It is an exemplary flow chart of a video generation method according to some embodiments of the present specification.
[0039] Figure 6a It is a schematic diagram of the first stage video generation method shown in some embodiments of this specification.
[0040] Figure 6b It is a schematic diagram of the second stage video generation method shown in some embodiments of this specification.
[0041] Figure 7 is an exemplary flow chart of another video generation method shown in some embodiments of this specification.
[0042] Figure 8 is an exemplary flow chart of another video generation method shown in some embodiments of this specification.
[0043] Fig. 9 is an exemplary flow chart of another video generation method shown in some embodiments of this specification.
[0044] Fig.10 Schematic diagram of the video frames generated by the video generation model in the first and second stages.
[0045] Fig.11 It is an exemplary block diagram of a training device for a video generation model according to some embodiments of the present specification.
[0046] Fig.12 is an exemplary block diagram of a video generating device according to some embodiments of the present specification. DETAILED DESCRIPTION
[0047] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the contents described below are some examples or embodiments of this specification. For ordinary technicians in this field, without paying creative work, the technical solutions or means disclosed in this specification can also be applied to other scenarios based on these technical contents.
[0048] It should be understood that the "system", "device", "unit" and / or "module" used in this specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.
[0049] Unless otherwise specified, technical terms used in this specification to describe components, elements, etc. do not refer to the singular, but may also include the plural. Generally speaking, terms such as "include", "comprise", etc. only indicate that the steps, elements, or components that have been clearly identified are included, and these steps, elements, and components do not constitute an exclusive list, such as the method or device described may also include other steps or components.
[0050] Flowcharts are used in this specification to illustrate the operation steps performed by the device or system of the relevant embodiments, but unless otherwise specified, the order used to describe these steps should not be understood as a limitation on the order in which the steps are performed. A person of ordinary skill in the art can adjust the order in which these steps are performed based on the knowledge and information conveyed by the embodiments of this specification, and the adjustment includes but is not limited to swapping the order, merging multiple steps, and splitting a certain step.
[0051] First, the terms related to some embodiments of this specification are explained.
[0052] Text / Image-to-Video Generation refers to the technology of converting text descriptions / images into video content.
[0053] Diffusion Models, a generative model based on the diffusion process, is used to generate images or videos from noise samples by gradual diffusion processing.
[0054] Stable Diffusion Model, a Vincent graph model based on the latent diffusion model.
[0055] Variational AutoEncoder (VAE) is a generative model that maps high-dimensional data to low-dimensional latent vectors through an encoder and reconstructs the low-dimensional latent vectors back to high-dimensional data through a decoder.
[0056] Text feature encoder is a technique that converts text data into numerical features so that it can be processed by machine learning models.
[0057] Structural Similarity Index (SSIM) is an evaluation index used to measure the similarity of images.
[0058] Generating videos based on text and images is a research direction that has attracted much attention and developed rapidly in the field of artificial intelligence in recent years. In order for computers to master the ability to generate videos based on text and images, it is necessary to build a corresponding video generation model to define the complex mapping relationship between text, images and videos. By iteratively calculating and training this video generation model, an optimized video generation model can be obtained, and in subsequent use, the corresponding video content can be accurately provided based on the input text and images. The following will introduce the application of generating videos based on text and images in combination with specific scenarios.
[0059] Figure 1 Schematic diagram of video generation model application scenarios according to some embodiments of this specification. Figure 1 As shown, the application scenario may include a model training device 110, a video generating device 120 and a terminal device 130. The model training device 110 and the video generating device 120 may exchange data with the terminal device 130 through a network.
[0060] The model training device 110 can be a server, and the server can be a single computer device, or a computing cluster composed of multiple computer devices, or a cloud server that provides cloud computing services. The model training device 110 can also be an electronic device such as a computer, a mobile phone, a tablet computer, an intelligent robot, or some other electronic device with strong computing power. The video generation model can be deployed in the model training device 110, and the model training device 110 undertakes the main computing tasks of the video generation model training. In some embodiments, the model training device 110 can be configured to obtain training samples, the training samples include a continuous sequence of pictures and a text description suitable for describing the picture sequence, determine reference information based on the picture sequence, and perform iterative training of the video generation model in the first and second stages based on the training samples and the reference information, thereby obtaining the video generation model of the first and second stages and the corresponding weights. The model training device 110 can also be configured with a corresponding storage device to store data related to the video generation model training method, such as storing training samples, parameters of the video generation model, and other data related to the video generation model training method.
[0061] The trained video generation model can be deployed in the video generation device 120 for use to provide the function of generating videos based on text and images. The video generation device 120 can be a server, which can be a single computer device, a computing cluster composed of multiple computer devices, or a cloud server that provides cloud computing services. The video generation device 120 can also be a terminal device such as a mobile phone, a computer, a tablet computer, a smart TV, a multimedia playback device, a wearable device, a vehicle terminal, a PC (Personal Computer), etc. The video generation device 120 can use local resources for reasoning, thereby providing efficient and fast video generation services. In some embodiments, the video generation device 120 can be configured to obtain input content, the input content includes an input reference image and a text description corresponding to the reference image, determine reference information based on the reference image, and input the input content and reference information into the pre-trained first-stage video generation model to obtain a first generated video, then obtain the selected first generated video, and then input the input content, reference information and the selected first generated video into the pre-trained second-stage video generation model to obtain a second generated video.
[0062] The terminal device 130 may be, for example, a mobile phone, a computer, a tablet computer, a smart TV, a multimedia player, a wearable device, a vehicle terminal, a PC (Personal Computer), etc. In the training phase, the terminal device 130 may transmit data to the model training device 110 to control and manage the model training process. In the model application phase, when the video generation device 120 is a server, the terminal device 130 may transmit data with the video generation device 120 to generate a video in response to user needs. For example, a user may input through an input device (such as a touch screen, keyboard, mouse, etc.) of the terminal device 130, and the terminal device 130 transmits the pictures and text descriptions input by the user to the video generation device 120, and the video generation device 120 generates a video based on the pictures and text descriptions, and transmits it to the terminal device 130, and the terminal device 130 presents the video to the user through a display device (such as a display screen).
[0063] This application can be applied to a variety of scenarios where video output is required, including but not limited to video content creation, film and television production, virtual reality, and social media. For example, a video clip can be generated based on a picture and text description input by the content creator. The content creator can make short videos, animations, or generate virtual anchors based on the video clips, thereby providing creative inspiration and tool support for content creators. In addition, the types of generated videos can include multiple types, including but not limited to portrait videos, landscape videos, and cartoon videos.
[0064] The video generation model in some embodiments of the present specification can be implemented based on the stable diffusion model. The core of stable diffusion is to gradually transform the simple initial distribution into a complex target distribution through a random process. It gradually adds noise to the data through the forward diffusion process until it becomes completely noisy data, and then gradually removes noise from the completely noisy data through the reverse diffusion process to restore the original data.
[0065] In some embodiments, the video generation model may include an encoding module and a backbone network including a timing module. The encoding module may include a variational autoencoder and a text feature encoder. The variational autoencoder is used to encode the input content, map high-dimensional data to low-dimensional potential vectors through mapping learning, and reconstruct the low-dimensional potential vectors back to high-dimensional data through a decoder. The text feature encoder is used to convert text data into feature vectors. The backbone network may be U-net, which is a convolutional neural network with a U-shaped structure, commonly used for content generation. Its main task is to gradually denoise based on the diffusion principle and recover high-quality images from noise. The structure of the backbone network (U-net) may include an encoder (downsampling path), a bottleneck layer, a decoder (upsampling path) and an output layer. The encoder is used to perform a series of convolution operations and pooling operations on the input tensor to extract the features of the image. Each layer includes two convolution layers, followed by a pooling layer. After each pooling layer, the image size is halved, but the number (depth) of feature maps increases. The bottleneck layer is used to connect the encoder and decoder parts and includes two convolution layers. The decoder restores the size of the image through upsampling and convolution operations, and combines the feature maps of the encoder part. Each layer contains an upsampling operation (transposed convolution) followed by two convolution operations. The upsampled feature map is spliced with the corresponding feature map of the encoder part to retain the detail information. Finally, the feature map is converted into the final result through the output layer. A timing module composed of a 3D neural network can be added between the encoder and decoder of the backbone network. The timing module is suitable for capturing the temporal correlation between images so as to generate multiple frames with continuity. In some embodiments, the variational autoencoder and the text feature encoder can also be replaced by other encoders or models with encoding functions, which are not specifically limited here.
[0066] In other embodiments, the video generation model may further include a content guide, which may include an image feature encoder and a mapping network composed of a fully connected network. The image feature encoder can extract the image feature information in the reference image and represent it as a vector of a specific dimension. The dimension of the image feature vector may be different from the dimension of the text feature vector, so it is necessary to map the image feature vector to the same dimension as the text feature vector through a mapping network. In some embodiments, the dimension of the image feature vector can be adjusted by linearly transforming the image feature vector through a fully connected network, and the linear transformation can be achieved through a weight matrix corresponding to the fully connected network. Exemplarily, the dimension of the image feature vector is d1, the dimension of the text feature vector is d2, and the weight matrix corresponding to the fully connected network can be d1×d2. Through the multiplication operation of the weight matrix, the dimension of the image feature vector can be converted from d1 dimension to d2 dimension. .
[0067] Figure 2 is an exemplary flow chart of a video generation model training method according to some embodiments of the present invention. The method can be Figure 1 The model training device 110 shown in FIG. 1 is executed, and the model training device 110 may include Fig.11 The framework shown. Figure 2 As shown, the method specifically includes the following contents.
[0068] S210: Obtain training samples.
[0069] Can be done by Fig.11 The acquisition module 1110 shown is used to acquire training samples.
[0070] In some examples, the training sample may include a continuous sequence of pictures and a text description suitable for describing the sequence of pictures. The continuous sequence of pictures may be a plurality of pictures arranged in time order, and there is a temporal sequence between adjacent pictures, for example, a sequence of frames constituting a video. The text description is suitable for explaining the content expressed by the sequence of pictures. The training sample may include one or more sets of paired picture sequences and text descriptions.
[0071] In some embodiments, the acquisition module 1110 can collect video data of a certain length from an open source data set or other video library, and then use a video slicing tool to cut the collected video data into multiple video clips that do not contain transition shots, and then use a video description model (or tool) to automatically describe the content of the cropped video clips, thereby obtaining paired video clips and text descriptions. Among them, the type of video data collected can be determined according to the type of video to be generated by the video generation model. For example, for a video generation model to generate a portrait video, the video data collected during the training phase can be portrait video data. The duration of each video clip can be set according to the video generation requirements, for example, to a few seconds. The video slicing tool can be, for example, a video scene detector, which can analyze the changes in the video screen. When an obvious scene transition is detected, the video is sliced at that time point. The video slicing tool can also be other video editing tools with scene detection functions, which are not specifically limited here. The video description model can be a multimodal large model, or it can be other video description models (or tools) with video description functions, which are not specifically limited here.
[0072] In some embodiments, after obtaining the video clip, the acquisition module 1110 may further uniformly extract preset frame images from the video clip as a continuous image sequence, and form a paired training sample with the corresponding text description. For example, for a certain video clip, n frames of images I may be uniformly extracted from the video clip, and form a paired training sample with the text description.
[0073] In some embodiments, the Fig.11 The preprocessing module 1150 shown is used to preprocess the pictures in the picture sequence so that the image resolution of the picture sequence in the second stage is greater than the image resolution of the picture sequence in the first stage, and in the subsequent process, the reference information determination module 1120 determines the reference information of the first stage and the second stage respectively based on the preprocessed picture sequence.
[0074] Preprocessing can include adjusting the resolution of each image in the image sequence and normalization. Specifically, the resolution of each image in the image sequence can be adjusted by downsampling or upsampling. Downsampling reduces the resolution of the original image by reducing the number of data points or pixels, thereby reducing the amount of image data and computational complexity. Upsampling increases the resolution of the original image by increasing the number of data points or pixels, thereby generating a clearer image and improving image quality. Normalization refers to a series of standard processing transformations on the image so that the preprocessed data is limited to a certain range.
[0075] For the picture sequence used in the first stage of model training, each picture in the picture sequence can be downsampled to reduce the resolution of the original picture, obtain a picture sequence of the first resolution, and train based on the picture sequence of the first resolution in the subsequent process, thereby reducing the amount of calculation and storage space of the first stage training, shortening the training cycle, and making the first stage video generation model training more focused on the overall framework. For the picture sequence used in the second stage of model training, each picture in the picture sequence can be downsampled (or upsampled) to obtain an image sequence of the second resolution, wherein the second resolution is greater than the first resolution, and train based on the image sequence of the second resolution in the subsequent process, so that the second stage training focuses more on capturing detailed features, training the video generation model from low resolution to high resolution, and also allowing the video generation model to gradually refine feature learning, strengthen the stability of the video generation model in the early stage of training, and enable the video generation model to achieve a better balance between global structure and local details.
[0076] In some embodiments, the preprocessing module 1150 may perform centralized preprocessing on the image sequence so as to obtain an image sequence of the first resolution and the second resolution at one time. In other embodiments, the preprocessing module 1150 may also perform preprocessing on the image sequence in the first stage and the second stage respectively, for example, preprocessing is performed once in the first stage to obtain an image sequence of the first resolution, and preprocessing is performed again in the second stage to obtain an image sequence of the second resolution, and the second resolution is higher than the first resolution. In some other embodiments, the second resolution is an integer multiple of the first resolution. For example, the second resolution is twice the first resolution. The steps of preprocessing each image in the image sequence can be flexibly adjusted in order of execution according to actual processing requirements. For example, the preprocessing module 1150 may perform preprocessing after obtaining the training sample, or may perform preprocessing after determining the reference image below, and this specification does not make specific limitations.
[0077] S220: Determine reference information according to the picture sequence.
[0078] Can be done by Fig.11 The reference information determination module 1120 shown determines reference information according to the picture sequence.
[0079] In some embodiments, the reference information may include a reference image related to the picture sequence and similarity information between the reference image and the picture sequence. In other embodiments, the reference information further includes detail information extracted from the reference image.
[0080] According to some embodiments, the reference information determination module 1120 may determine any one of the pictures in the picture sequence as a reference picture. The reference picture may provide a clear reference basis for the video generation model, thereby helping the video generation model to generate a more accurate image sequence. Figure 2a The schematic diagram of the video generation model in the first stage training method is shown in FIG. i (i=1,2……n), the k∈[0,n)th frame can be randomly selected from n frames as the reference image I k .
[0081] According to some embodiments, the reference information determination module 1120 may extract high-frequency details in the reference image as detail information through an edge detection operator. The edge detection operator may be, for example, a Sobel operator, which is a commonly used image processing technique for edge detection. It determines the position of the edge by calculating the gradient value of each pixel in the image to extract high-frequency signals in the image, thereby increasing the details and features of the generated content and more clearly displaying the outline, shape and boundary of the object, such as clearly displaying the eyes, hair, clothing patterns, etc. of a person. For example, Figure 2a As shown in reference figure I k , the reference image I can be extracted by Sobel operator k High frequency details in I k s In some other embodiments, other edge detection operators may be used to extract high-frequency details in the reference image, which is not specifically limited here. By extracting the detail information of the reference image, the video generation model can better capture and learn the detail features in the image, enhance the detail expression in the generated video, and improve the quality of the generated video.
[0082] In some embodiments, it is also possible to Fig.11 The encoding module 1161 shown encodes the reference image and detail information in the training sample and reference information for subsequent calculation or input into the backbone network of the video generation model. Specifically, the encoding module 1161 may include a variational autoencoder and a text feature encoder. The variational autoencoder is suitable for encoding the image sequence, the reference image and the high-frequency details into a latent vector, and the text feature encoder is suitable for encoding the text description into a text feature vector. For example, Figure 2a As shown, the n-frame image I can be transformed into i Encoded as n latent vectors L i , (i=1,2……n), refer to Figure I k Encoded as the reference image latent vector L k , the high frequency details I k sEncoded as a high-frequency detail latent vector L k s , the text description is encoded into a text feature vector C through text feature encoding.
[0083] In some embodiments, the reference information determination module 1120 can calculate the similarity coefficient between the reference image and each image in the image sequence based on the structural similarity. By calculating the similarity coefficient between the reference image and each image in the image sequence, the amplitude of movement of the person or object in the image sequence can be determined. For example, for a portrait video, the amplitude of movement of the person can be determined, and for a non-portrait video, the amplitude of movement or change of the object can be determined. The lower the similarity between the reference image and the image in the image sequence, the greater the amplitude of movement of the person or object, and the higher the similarity, the smaller the amplitude of movement of the person or object.
[0084] Specifically, the reference information determination module 1120 can calculate the similarity coefficient based on the encoded picture sequence and the reference picture. For example, for the n latent vectors L encoded by the variational autoencoder i and the reference image latent vector L k , the latent vector L of each frame can be calculated by the following formula i With the reference image latent vector L k The similarity coefficient A i :
[0085] A i =1-(1-SSIM(Norm(L i ),Norm(L k )))*α
[0086] Where i = 1, 2, ... n, Norm (L i ) and Norm(L k ) represents the latent vector L i and the latent vector L k Normalization is performed, SSIM(Norm(L i ),Norm(L k ) represents the calculation of the normalized latent vector L i and L k The structural similarity index between them is α, which is an adjustable hyperparameter.
[0087] By determining the change in similarity between the reference image and each image in the image sequence, the motion change of the person or object in the image sequence can be determined. For example, in a set of similarity coefficients, A i When the change is from small to large, it means that the movement amplitude of the person or object in the picture sequence is getting smaller and smaller; A i The change from large to small indicates that the movement of the person or object in the picture sequence is getting larger and larger; A iThe change is from small to large and then to small, which means that the movement of the person or object in the picture sequence is a cyclic motion, such as the person waving his hands or shaking his head back and forth in the picture sequence.
[0088] By calculating the similarity coefficient between the reference image and the image in the image sequence, and inputting it into the video generation model for iterative training, when the video generation model is used subsequently, the relevant similarity coefficient sequence is matched based on the image and text description input by the user, so as to accurately control the motion amplitude in the generated video picture, and realize accurate control of the video generated content through input. The similarity coefficient can also provide an explicit guiding signal for the video generation model to enhance the stability of video generation and improve the consistency between the generated video and the input content.
[0089] In some embodiments, before iteratively training the video generation model, the following Fig.11 The first stage training module 1130 shown first loads the pre-trained weights of the video generation model, and freezes the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network, that is, the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network remain unchanged during the training process, avoiding being updated during the training process, thereby not destroying the original structure of the video generation model, so that the video generation model can easily load the model weights trained by other video generation models to transform the style of the generated video, and has good scalability. The step of loading the pre-trained weights of the video generation model can also be performed before preprocessing the image sequence, which is not specifically limited here.
[0090] S230, iteratively train the video generation model according to the training samples and the reference information to obtain a first-stage video generation model and a corresponding first model weight.
[0091] Can be done by Fig.11 The first stage training module 1130 shown performs the first stage iterative training on the video generation model.
[0092] Iterative training is a training method in machine learning and deep learning. It refers to training and optimizing the model through multiple cycles (iterations) to gradually improve the model and eventually achieve the desired performance. Each cycle is called an iteration. In each iteration, the model uses a portion of the training data for forward propagation and back propagation to adjust the model parameters.
[0093] Model weights refer to the parameters learned in a machine learning or deep learning model. During the training process, the model adjusts these weights by learning the mapping relationship from input to output so that the model can predict the target value as accurately as possible.
[0094] Specifically, if Figure 3As shown, step S230 may include the following contents.
[0095] S310 , performing noise processing on the encoded picture sequence to obtain a noisy image and input noise.
[0096] In some embodiments, the first stage training module 1130 may first determine the type of noise to be added, such as Gaussian noise, and then add noise to the latent vector corresponding to each image in the image sequence. Figure 2a As shown, based on the diffusion model principle, n latent vectors L can be 1、 L 2... L n Add noise at any time to obtain the noisy image L t and input noise N t .
[0097] S320, inputting the noisy image, similarity information, the encoded reference image, the encoded detail information, the encoded text description and the preset timestamp into the backbone network.
[0098] The preset timestamp means that in the denoising process of the diffusion model, each sampling step has a corresponding timestamp, that is, each iteration corresponds to a time step, which is used to guide each step of the denoising process.
[0099] In some embodiments, the first stage training module 1130 may first concatenate the latent vectors corresponding to the noisy image and the reference information. For example, Figure 2a As shown, the noisy image L t With the reference image latent vector L k , high-frequency detail latent vector L k s and similarity coefficient A i The spliced information is then input together with the timestamp t and the text feature vector C into Fig.11 The backbone network 1163 of the video generation model 1160 is shown.
[0100] S330: Obtain output noise output by the backbone network corresponding to a preset timestamp.
[0101] Specifically, the backbone network outputs the output noise corresponding to the current time step during each iteration. These output noises are used to guide the model to gradually denoise during the generation process, restoring from pure noise to clear picture sequences, thereby obtaining clear video clips.
[0102] S340, adjusting the weight of the first-stage video generation model according to the input noise and the output noise.
[0103] Specifically, loss parameter information can be determined according to the input noise and the output noise, and the weight of the video generation model can be adjusted according to the loss parameter information.
[0104] In some embodiments, the loss parameter information can be calculated by a loss function formula, for example, Figure 2a As shown, for the input noise N t and output noise N t ’ , the loss Loss can be calculated by the following loss function formula:
[0105] Loss = ||N t '-N t || 2
[0106] In some embodiments, the first stage training module 1130 can also adjust the weights of the unfrozen input layer and the weights of the timing module in the backbone network according to the loss parameter information. Specifically, the unfrozen input layer and the timing module in the backbone network can be optimized by back-propagating the loss through the gradient descent method. The gradient descent method is an iterative method for optimization problems. It minimizes the value of the loss function (or objective function) by continuously adjusting the parameters. Its core is to move the parameters in the opposite direction of the gradient according to the gradient of the current parameters to find the minimum value of the loss function. Back propagation is a formalized gradient descent algorithm that can propagate forward from the output layer step by step along the hierarchy of the neural network to calculate the parameter gradients of each layer. The smaller the loss Loss, the more accurate the backbone network's prediction of noise.
[0107] Repeat S310 to S340 for iterative training to obtain the first-stage video generation model and the corresponding first model weights, thereby completing the first-stage training. The first model weights refer to all parameters used in the neural network to generate the video.
[0108] When performing the second stage of training, the resolution of the images in the image sequence can be sampled to a second resolution (the second resolution is greater than the first resolution), or a sequence of images with the second resolution that has been pre-processed in the preprocessing stage can be directly obtained, and then reference information is determined based on the image sequence with the second resolution and used in the second stage of training below (the method of preprocessing and determining reference information is similar to the above, and for details, please refer to the description in S210 and S220 above).
[0109] For example, see Figure 2b The schematic diagram of the video generation model in the second stage training method is shown, for the second resolution image sequence I 1、 I 2... I n, we can randomly select k∈[0,n) frames from n frames as reference images I k , extract the reference image I through the sobel operator k High frequency details in I k s , through the variational autoencoder, n frames of images I i (i=1,2……n) is encoded into n latent vectors L i , (i=1,2……n), refer to Figure I k Encoded as the reference image latent vector L k , the high frequency details I k s Encoded as a high-frequency detail latent vector L k s , the text description is encoded into text features C through the text feature encoder, and the latent vector L of each frame is calculated i With the reference image latent vector L k The similarity coefficient A i , thereby obtaining reference information including reference image latent vector, similarity information and detail information latent vector, which is used in the following training process.
[0110] S240: Using the first model weight as the initial weight, and iteratively training the video generation model obtained in the first stage according to the training samples and the reference information to obtain the video generation model of the second stage and the corresponding second model weight.
[0111] The resolution of the picture sequence and the reference picture in the second stage is greater than that of the picture sequence and the reference picture in the first stage.
[0112] Can be done by Fig.11 The second stage training module 1140 shown performs second stage iterative training on the video generation model. Specifically, the second stage training module 1140 can first load the first model weight obtained in the first stage as the initial weight for training the second stage video model, and perform noise processing on the encoded picture sequence to obtain a noisy image and input noise, splice the noisy image with the similarity information, the encoded reference image, and the encoded detail information, and input them into the backbone network together with the encoded text description and the preset timestamp, obtain the output noise corresponding to the preset timestamp output by the backbone network, and adjust the weight of the second stage video generation model according to the input noise and the output noise, repeat the process for iterative training, so as to obtain the second stage video generation model and the corresponding second model weight (the above process is similar to the previous text, and the specific description of S310-S340 in the previous text can be referred to, which will not be repeated here).
[0113] In some embodiments, during the second stage of training, the video generation model may further include a content guidance module to generate content guidance information to further describe the details of the generated video. When the content guidance information is added, Figure 4 As shown, S240 may specifically include the following contents.
[0114] S410 . Determine content guidance information through a content guidance module, where the content guidance information is determined based on a reference image and a text description.
[0115] Specifically, the content guidance module may include a pre-trained image feature encoder and a mapping network composed of a fully connected network. The image feature information in the reference image is extracted through the image feature encoder, and the image feature vector is mapped to the same dimension as the text feature vector through the mapping network. Then, the image feature information of the same dimension is concatenated with the text description to obtain content guidance information. Among them, the image feature information is suitable for expressing the content presented in the reference image. Further, the image feature information can represent complex concepts and objects, and capture the semantic information present in the image, such as object category, action, scene type, etc. It can also be combined with the contextual information in the image to understand the association between objects and the overall meaning of the scene.
[0116] For example, if Figure 2b As shown in reference figure I k and text features C, extracted from the reference image I by the image feature encoder k Image feature information C_I K , and the image feature information C_I is mapped through the network K Mapped to the same dimension as the text feature C, the image feature information C_I K Combined with text feature C, content guidance information is obtained.
[0117] By adding content guidance information in the second stage, the details in the picture can be further captured and the detail depiction can be enhanced to generate more realistic and vivid video images.
[0118] S420: Use the first model weight as the initial weight, and iteratively train the video generation model obtained in the first stage according to the training samples, reference information, and content guidance information to obtain the video generation model of the second stage and the corresponding second model weight.
[0119] The resolution of the picture sequence and the reference picture in the second stage is greater than that of the picture sequence and the reference picture in the first stage.
[0120] Specifically, if Figure 5 As shown, step S420 may include the following contents.
[0121] S510: Perform noise processing on the encoded picture sequence to obtain a noisy image and input noise.
[0122] The resolution of the pictures in the picture sequence is the second resolution, which is greater than the second resolution of the pictures in the picture sequence in the first stage.
[0123] In some embodiments, the second stage training module 1140 may first determine the type of noise to be added, such as Gaussian noise, and then add noise to each image in the image sequence at any time. Figure 2b As shown, based on the diffusion model principle, n latent vectors L can be 1、 L 2... L n Add noise at any time to obtain the noisy image L t and input noise N t .
[0124] S520: input the noisy image, similarity information, encoded reference image, encoded detail information, content guide information and preset timestamp into the backbone network.
[0125] The reference information is obtained based on a picture sequence of the second resolution.
[0126] Specifically, the second stage training module 1140 may first concatenate the latent vectors corresponding to the noisy image and the reference information, for example, Figure 2b As shown in Figure 2, the noisy image L t With the reference image latent vector L k , high-frequency detail latent vector L k s and similarity coefficient A i splicing, and combined with the image feature information C_I K The content guidance information concatenated with the text feature C and the timestamp t are input into Fig.11 The backbone network 1163 of the video generation model 1160 is shown.
[0127] S530: Obtain output noise corresponding to a preset timestamp output by the backbone network.
[0128] This process is similar to the previous one. For details, please refer to the description of S330 in the previous article, which will not be repeated here.
[0129] S540: Adjust the weight of the second-stage video generation model according to the input noise and the output noise.
[0130] This process is similar to the process of S340 in the previous text. The difference is that when adjusting the weights of the second-stage video generation model according to the loss parameter information, the weights of the unfrozen input layer in the backbone network, the weights of the timing module, and the weights of the content guide can be adjusted. For other descriptions, please refer to the description of S340 in the previous text, which will not be repeated here.
[0131] S510 to S540 are repeated to perform iterative training to obtain the video generation model of the second stage and the corresponding second model weights, thereby completing the second stage training. The second model weights refer to all parameters used in the neural network to generate the video.
[0132] The video generation model training method provided in some embodiments of the present specification has the following beneficial effects: (1) by adding reference images and similarity information between reference images and image sequences to the input, the video generation model can learn various motion changes of people or objects in the training samples, enhance the stability of video generation in subsequent stages, improve the consistency between the generated video and the input content, and enhance the coherence of the generated video images; (2) by adding detailed information of the reference images to the input of the two stages, the video generation model can better capture and learn the detailed features in the image, and enhance the detailed expression in the generated video; (3) the resolution of the images in the training samples of the second stage is greater than the resolution of the images in the training samples of the first stage, which greatly optimizes the computing power usage and enables the video generation model to learn with different emphases at different stages, gradually refine the features on the basis of a stable training process, and improve the training efficiency; (4) The stage-cascaded model architecture enables the video generation model to maintain the stability of the generated video while obtaining higher resolution and clarity. It also well maintains most of the details of the input image and meets the requirements for details and realism. (5) By freezing the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network, only the weights of the unfrozen input layer, timing module and content guide are trained, which makes the training process shorter and greatly saves computing resources. At the same time, the original structure of the video generation model is not destroyed, so that the video generation model can easily load the trained model weights of other video generation models to change the style of the generated video, and has good scalability. (6) Text descriptions are injected in both training stages, and the content guide is introduced in the second stage, so that the video generation model has strong semantic controllability, and users can achieve accurate and reasonable motion generation through text.
[0133] Figure 6 is an exemplary flow chart of a video generation method according to some embodiments of the present invention. The method can be Figure 1 The video generating device 120 shown in FIG. 1 is executed, and the video generating device 120 may include Fig.12 The framework shown. Figure 6 As shown, the method specifically includes the following contents.
[0134] S610: Obtain input content.
[0135] Can be done by Fig.12 The content acquisition module 1210 shown is used to acquire training samples.
[0136] In some embodiments, the input content includes an input reference image and a text description corresponding to the reference image. Figure 1 The terminal device 130 shown is used to input a picture and make a text description of the desired video to be generated. The content acquisition module 1210 directly obtains the picture (i.e., the reference picture) and the corresponding text description, or the terminal device 130 obtains the content input by the user and transmits it to the video generation device 120, and then the content acquisition module 1210 in the video generation device 120 obtains the picture (i.e., the reference picture) and the corresponding text description.
[0137] In some embodiments, the Fig.12 The preprocessing module 1260 shown is used to preprocess the pictures in the picture sequence so that the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model, and in the subsequent process, the reference information determination module 1220 determines the reference information input into the first-stage video generation model and the second-stage video generation model respectively based on the preprocessed reference image.
[0138] Preprocessing may include cropping, downsampling and normalization of the reference image. Cropping refers to adjusting the size of the image to meet the requirements of subsequent image processing. The description of downsampling and normalization can be found in the description of S210 above, which will not be repeated here.
[0139] For the reference image used in the first stage of video generation, the original reference image can be downsampled to reduce the resolution of the original reference image, obtain a reference image of a third resolution, and generate the first stage video based on the reference image of the third resolution in the subsequent process, thereby reducing the computational complexity and storage space of the first stage video generation. For the reference image used in the second stage of video generation, the original reference image can be downsampled (or upsampled) to obtain a fourth resolution image, wherein the fourth resolution is greater than the third resolution, and the second stage video is generated based on the reference image of the fourth resolution, thereby outputting high-quality video in the second stage of video generation. The method of outputting the low-resolution generation result first and then the high-resolution generation result greatly optimizes the computing power occupation and realizes more efficient and lower-cost video generation. In some embodiments, the third resolution may be the same as the first resolution or different from the first resolution; the fourth resolution may be the same as the second resolution or different from the second resolution. In other embodiments, the difference between the third resolution and the first resolution (or the fourth resolution and the second resolution) is less than a certain set range.
[0140] In some embodiments, the preprocessing module 1260 can perform centralized preprocessing on the reference images so as to obtain reference images of the third resolution and the fourth resolution at one time. In other embodiments, the preprocessing module 1260 can also preprocess the reference images in the first stage and the second stage respectively, for example, preprocessing is performed once in the first stage to obtain a reference image of the third resolution, and preprocessing is performed again in the second stage to obtain a reference image of the fourth resolution. The fourth resolution is higher than the third resolution. In some other embodiments, the fourth resolution is an integer multiple of the third resolution. Exemplarily, the fourth resolution is twice the third resolution. The steps of preprocessing the reference images can be flexibly adjusted in the order of execution according to actual processing requirements, and are not specifically limited here.
[0141] S620: Determine reference information according to the reference graph.
[0142] Can be done by Fig.12 The reference information determination module 1220 shown determines reference information according to the reference graph.
[0143] In some embodiments, the reference information may include similarity information related to the input content. In other embodiments, the reference information may also include detail information extracted from the reference graph.
[0144] According to some embodiments, the reference information determination module 1220 can extract high-frequency details in the reference image as detail information. The specific method of extracting high-frequency details in the reference image is similar to that in the training phase. For details, please refer to the description in S220 above, which will not be repeated here.
[0145] For example, see Figure 6a The schematic diagram of the first stage video generation method shown in FIG. k , the high-frequency details in the reference image can be extracted by the Sobel operator I k s .
[0146] According to some embodiments, the reference information determination module 1220 may generate a similarity coefficient related to the input content through a preset periodic function, and the preset periodic function may be, for example, a trigonometric function. Figure 6a As shown, a string of similarity coefficients A can be simulated by the similarity coefficient sequence generation function i .
[0147] During the training phase, the video generation model learns and trains a large number of similarity coefficient sequences corresponding to different motion conditions. During the video generation phase, the video generation model can generate a string of similarity coefficients that meet the requirements based on the requirements for motion conditions in the input reference images and text descriptions as well as some preset business requirements through a preset periodic function, and guide video generation based on the similarity coefficient sequence, thereby accurately controlling the motion amplitude in the generated video screen, and realizing precise control of video generated content through input. The similarity coefficient can also provide an explicit guiding signal for the video generation model, enhance the stability of video generation, and improve the consistency between the generated video and the input content.
[0148] In some embodiments, it is also possible to Fig.12 The encoding module 1271 shown encodes the reference image, text description and detail information for subsequent input into the backbone network of the video generation model. Specifically, the encoding module 1271 may include a variational autoencoder and a text feature encoder. The variational autoencoder is suitable for encoding the image sequence, the reference image and the high-frequency details into a latent vector, and the text feature encoder is suitable for encoding the text description into a text feature vector. For example, Figure 6a As shown, the reference image I is transformed by the variational self-encoder k Encoded as the reference image latent vector L k , the high frequency details I k s Encoded as a high-frequency detail latent vector L k s , the text description is encoded into a text feature vector C through text feature encoding.
[0149] Before the input content and reference information are fed into the backbone network of the pre-trained first-stage video generation model, Fig.12The first video generation module 1230 shown is used to load the pre-trained first-stage video generation model and the corresponding first model weights. The step of loading the pre-trained first-stage video generation model can also be performed before preprocessing the reference image, which is not specifically limited here.
[0150] S630: Input the input content and reference information into a pre-trained first-stage video generation model to obtain a first generated video.
[0151] Can be done by Fig.12 The first video generation module 1230 shown inputs the input content and reference information into the pre-trained first stage video generation model to obtain a first generated video.
[0152] Specifically, if Figure 7 As shown, step S630 may include the following contents.
[0153] S710: Obtain two or more preset noises that are consistent with the dimensions of the encoded reference image.
[0154] In some embodiments, the first video generation module 1230 may randomly initialize n latent vectors L corresponding to the reference image. k The preset noise has the same dimension so that the backbone network can gradually denoise based on the noise information. The preset noise can be Gaussian noise.
[0155] S720: input two or more preset noises, the encoded reference image, the encoded detail information, the encoded text description, the similarity information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a first image sequence latent vector.
[0156] Specifically, the first video generation module 1230 can first splice the two or more initialized preset noises with the encoded reference image, the encoded detail information and the similarity information, and then input the spliced information, the preset timestamp and the encoded text description into the backbone network for iterative denoising to obtain the first image sequence latent vector output by the backbone network.
[0157] For example, if Figure 6b As shown, the first video generation module 1230 can firstly combine n Gaussian noises with the reference image latent vector L k , high-frequency detail latent vector I k s And the similarity coefficient A i Then the spliced information is input together with the timestamp t and the text feature vector C into Fig.12 Iterative denoising is performed in the backbone network 1273 in the video generation model 1270 shown in the figure to obtain n image sequence latent vectors M output by the backbone network 1273 1、M 2... M n .
[0158] S730 . Decode the first image sequence latent vector through an encoding module to obtain a first generated video.
[0159] For example, if Figure 6b As shown, the n image sequence latent vector M can be transformed into 1、 M 2... M n Decoded into n frames of video m1, m2…m n , that is, the first generated video.
[0160] In some embodiments, before obtaining the selected first generated video, the first video generation module 1230 may also change the random seed and generate more than two first generated videos. The random seed is used to generate random numbers in a computer program to control the randomness of the program. Since the noise addition process is random, changing the random seed can obtain different generation results.
[0161] Specifically, the first video generation module 1230 can replace the random seed P times (P≥2) times, and repeat the steps after obtaining the input content in the above process (i.e., centralized preprocessing of the image sequence → determining the reference information → encoding through the encoding module → inputting the input content and the reference information into the pre-trained first-stage video generation model to obtain the first generated video) P times, obtain P first generated videos, and then display the P first generated videos to the user through the display device, or transmit them to the terminal device 130, and the terminal device 130 displays them to the user for the user to select.
[0162] S640: Obtain the selected first generated video.
[0163] When the user selects a first generated video, the video acquisition module 1240 correspondingly acquires the selected first generated video, or the terminal device 130 transmits the first generated video selected by the user to the video generating device 120, and the video acquisition module 1240 of the video generating device 120 acquires the selected first generated video.
[0164] In some embodiments, after the first video is generated, the resolution of the reference image can be sampled to a fourth resolution (the fourth resolution is greater than the third resolution), or a reference image with a fourth resolution that has been pre-processed in the preprocessing stage can be directly obtained, and then reference information is determined based on the reference image with the fourth resolution, and used for the video generation process in the second stage below (the method of preprocessing and determining reference information is similar to the above, and for details, please refer to the description in S610 and S620 above).
[0165] See Figure 6b The schematic diagram of the second stage video generation method shown in FIG. 1 for the fourth resolution k , the reference image I can be extracted by the sobel operator k High frequency details in I k s , through the variational autoencoder, the reference image I k Encoded as the reference image latent vector L k , the high frequency details I k s Encoded as a high-frequency detail latent vector L k s , the text description is encoded into text feature C through the text feature encoder, and the similarity coefficient sequence generation function generates the similarity coefficient A i , thereby obtaining reference information containing similarity information and detail information latent vectors, which are used in the video generation process in the second stage below.
[0166] S650, input the input content, reference information and the selected first generated video into a pre-trained second-stage video generation model to obtain a second generated video.
[0167] Among them, the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
[0168] Can be done by Fig.12 The second video generation module 1250 shown inputs the input content, reference information and the selected first generated video into the pre-trained second-stage video generation model to obtain the second generated video. Specifically, the second video generation module 1250 can first load the pre-trained second-stage video generation model and the corresponding second model weights, and then add noise to the first image sequence latent vector corresponding to the first generated video selected by the user at time t based on the diffusion model principle to obtain a noisy image, and input the noisy image and the encoded reference image, the encoded detail information, the encoded text description, the similarity information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a second image feature sequence, and decode the second image feature sequence through the encoding module to obtain the second generated video.
[0169] In some embodiments, when performing the second stage of video generation, the video generation model may further include a content guidance module to generate content guidance information to further describe the details of the generated video. When the content guidance information is added, such as Figure 8 As shown, S650 may specifically include the following contents.
[0170] S810. Determine content guidance information through a content guidance module, where the content guidance information is determined based on a reference image and a text description.
[0171] For example, if Figure 6b As shown in reference figure I k and text features C, extracted from the reference image I by the image feature encoder k Image feature information C_I K , and the image feature information C_I is mapped through the network K Mapped to the same dimension as the text feature C, the image feature information C_I K Combined with text feature C, content guidance information is obtained.
[0172] The rest of the description of this step is similar to the previous one. For details, please refer to the description in S410 above, which will not be repeated here.
[0173] S820: Input the input content, reference information, content guide information, and the selected first generated video into a backbone network for iterative noise reduction processing to obtain a second generated video.
[0174] Specifically, if Fig. 9 As shown, step S820 may include the following contents.
[0175] S910: Add noise based on the selected first generated video to obtain a noisy image sequence.
[0176] In some embodiments, the second video generation module 1250 can add noise to the first image sequence latent vector corresponding to the first generated video selected by the user at time t based on the diffusion model principle to obtain a noisy image sequence. Figure 6b As shown, the first image sequence latent vector M corresponding to the first generated video selected by the user can be 1、 M 2... M n Add noise at time t to obtain the noisy image sequence M t .
[0177] S920: Input the noisy image sequence, the encoded reference image, the encoded detail information, the similarity information, the content guide information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a second image feature sequence.
[0178] Specifically, the second video generation module 1250 can first splice the noisy image sequence with the encoded reference image, the encoded detail information and the similarity information, and then input the spliced information, content guide information and preset timestamp into the backbone network for iterative denoising to obtain the second image sequence latent vector output by the backbone network.
[0179] For example, if Figure 6b As shown, the first video generation module 1230 can first convert the noisy image sequence M t With the reference image latent vector L k , high-frequency detail latent vector I k s And the similarity coefficient A i Then combine the information after splicing with the content guidance information (image feature information C_I K The information concatenated with the text feature C) and the timestamp t are input into Fig.12 Iterative denoising is performed in the backbone network 1072 in the video generation model 1070 shown in the figure to obtain n image sequence latent vectors M' output by the backbone network 1072 1、 M' 2... M' n .
[0180] S930. Decode the second image feature sequence through the encoding module to obtain a second generated video.
[0181] For example, if Figure 6b As shown, the n image sequence latent vector M' can be transformed into 1、 M' 2... M' n Decode into n frames of video m' 1、 m' 2... m' n , that is, the second generated video.
[0182] In some embodiments, Fig.10 As shown, the first stage can generate video frames with lower resolution, thereby reducing the usage of computing resources and quickly and efficiently providing preliminary generation results for users to select. The video frames generated in the second stage have high resolution, clearer facial features and expressions, and richer details, thereby generating more vivid, clearer and high-quality videos.
[0183] The video generation method provided in some embodiments of the present specification has the following beneficial effects: (1) by determining similarity information to guide video generation, the motion amplitude in the generated video screen is accurately controlled, the stability of video generation is enhanced, and the consistency between the generated video and the input content is improved; (2) by determining the detail information of the reference image, the detail expression in the generated video can be enhanced; (3) the two-stage model allows the user to filter based on the first-stage generation results with lower resolution, and only perform the second-stage high-resolution generation on the selected results, which greatly optimizes the computing power usage. More efficient and lower-cost video generation is achieved; (4) the two-stage cascade model architecture enables the video generation model to maintain the stability of the generated video while obtaining a higher resolution and clarity generation result, well maintaining most of the details of the input image, and meeting the requirements for details and realism; (5) text descriptions are injected into both stages of video generation, and a content guidance module is introduced in the second stage, so that the video generation model has strong semantic controllability, and users can achieve accurate and reasonable motion generation through text.
[0184] Some embodiments of this specification also provide a training device for a video generation model. Fig.11 is an exemplary block diagram of a training device for a video generation model according to some embodiments of this specification, such as Fig.11 As shown, the training device includes the following modules.
[0185] The acquisition module 1110 is used to acquire training samples, where the training samples include a continuous picture sequence and a text description suitable for describing the picture sequence.
[0186] The reference information determination module 1120 is used to determine reference information according to the picture sequence, where the reference information includes reference images related to the picture sequence and similarity information between the reference images and the picture sequence.
[0187] The first stage training module 1130 is used to iteratively train the video generation model according to the training samples and reference information to obtain the first stage video generation model and the corresponding first model weight.
[0188] The second stage training module 1140 is used to use the first model weight as the initial weight, and iteratively train the video generation model obtained in the first stage according to the training samples and reference information to obtain the second stage video generation model and the corresponding second model weight, wherein the resolution of the picture sequence and the reference picture in the second stage is greater than the resolution of the picture sequence and the reference picture in the first stage.
[0189] In some optional embodiments, the reference information determination module 1120 is further used to determine any one picture in the picture sequence as a reference picture, extract high-frequency details in the reference picture, and calculate the similarity coefficient between the reference picture and each picture in the picture sequence based on structural similarity.
[0190] In some optional embodiments, a preprocessing module 1150 is further included for preprocessing the picture sequence so that the image resolution of the picture sequence in the second stage is greater than the image resolution of the picture sequence in the first stage. The reference information determination module 1120 is also used to determine the reference information of the first stage and the second stage respectively based on the preprocessed picture sequence.
[0191] In some optional embodiments, the video generation model 1160 includes an encoding module 1161 and a backbone network 1163 including a timing module 1162, wherein the encoding module 1161 is suitable for encoding training samples, reference images and detail information, the backbone network 1163 is suitable for predicting noise distribution according to an input noisy image based on the diffusion principle, and the timing module 1162 is suitable for capturing the temporal correlation between images.
[0192] In some optional embodiments, the reference information determination module 1120 is further configured to determine similarity information according to the encoded reference image and the encoded picture sequence.
[0193] In some optional embodiments, the first-stage training module 1130 is also used to perform noise processing on the encoded image sequence to obtain a noisy image and input noise; input the noisy image, similarity information, encoded reference image, encoded detail information, encoded text description and a preset timestamp into the backbone network; obtain the output noise corresponding to the preset timestamp output by the backbone network; adjust the weight of the first-stage video generation model according to the input noise and the output noise; and repeat the above process for iterative training.
[0194] In some optional embodiments, the first stage training module 1130 is further used to determine loss parameter information based on the input noise and the output noise; and adjust the weight of the video generation model based on the loss parameter information.
[0195] In some optional embodiments, the first stage training module 1130 is also used to freeze the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network; and adjust the weights of the unfrozen input layer in the backbone network and the weights of the timing module according to the loss parameter information.
[0196] In some optional embodiments, the video generation model also includes a content guidance module 1164, and the second stage training module 1140 is also used to determine content guidance information through the content guidance module 1164, and the content guidance information is determined based on the reference image and the text description; the first model weight is used as the initial weight, and the video generation model obtained in the first stage is iteratively trained according to the training samples, reference information and content guidance information to obtain the video generation model of the second stage and the corresponding second model weight.
[0197] In some optional embodiments, the second stage training module 1140 is also used to extract image feature information in the reference image through the content guidance module 1164, where the image feature information is suitable for expressing the content presented in the reference image; map the image feature information to the same dimension as the encoded text description; and concatenate the image feature information of the same dimension with the text description to obtain content guidance information.
[0198] In some optional embodiments, the second-stage training module 1140 is also used to perform noise processing on the encoded image sequence to obtain a noisy image and input noise; input the noisy image, similarity information, encoded reference image, encoded detail information, content guide information and a preset timestamp into the backbone network; obtain the output noise corresponding to the preset timestamp output by the backbone network; adjust the weight of the second-stage video generation model according to the input noise and the output noise; and repeat the above process for iterative training.
[0199] Some embodiments of the present specification also provide a video generating device. Fig.12 is an exemplary block diagram of a video generating device according to some embodiments of this specification, such as Fig.12 As shown, the training device includes the following modules.
[0200] The content acquisition module 1210 is used to acquire input content, where the input content includes an input reference image and a text description corresponding to the reference image.
[0201] The reference information determination module 1220 is used to determine reference information according to the reference graph, where the reference information includes similarity information related to the input content.
[0202] The first video generation module 1230 is used to input the input content and reference information into the pre-trained first-stage video generation model to obtain a first generated video.
[0203] The video acquisition module 1240 is used to acquire the selected first generated video.
[0204] The second video generation module 1250 is used to input the input content, reference information and the selected first generated video into a pre-trained second-stage video generation model to obtain a second generated video, wherein the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
[0205] In some optional embodiments, the reference information determination module 1220 is further configured to extract high-frequency details in the reference image; and generate a similarity coefficient related to the input content through a preset periodic function.
[0206] In some optional embodiments, the video generation device also includes a preprocessing module 1260, which is used to preprocess the reference image so that the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model; the reference information determination module 1220 determines the reference information input into the first-stage video generation model and the second-stage video generation model respectively based on the preprocessed reference image.
[0207] In some optional embodiments, the first video generating module 1230 is further used to replace the random seed and generate more than two first generated videos.
[0208] In some optional embodiments, the video generation model 1270 includes an encoding module 1271 and a backbone network 1273 including a timing module 1272, the encoding module 1271 is suitable for encoding input content and detail information, and decoding image features output by the backbone network, the backbone network 1273 is suitable for iteratively denoising the input noisy image based on the diffusion principle and outputting image features, and the timing module 1272 is suitable for capturing the temporal correlation between images.
[0209] In some optional embodiments, the first video generation module 1230 is also used to obtain two or more preset noises that are consistent with the dimensions of the encoded reference image; input the two or more preset noises, the encoded reference image, the encoded detail information, the encoded text description, the similarity information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a first image sequence latent vector; and decode the first image sequence latent vector through the encoding module to obtain a first generated video.
[0210] In some optional embodiments, the second-stage video generation model also includes a content guidance module 1274, and the second video generation module 1250 is also used to determine content guidance information through the content guidance module 1274, and the content guidance information is determined based on the reference image and the text description; the input content, reference information, content guidance information and the selected first generated video are input into the backbone network for iterative noise reduction processing to obtain the second generated video.
[0211] In some optional embodiments, the second video generation module 1250 is also used to extract image feature information in the reference image through the content guidance module 1274, where the image feature information is suitable for expressing the content presented in the reference image; map the image feature information to the same dimension as the encoded text description; and concatenate the image feature information of the same dimension with the text description to obtain content guidance information.
[0212] In some optional embodiments, the second video generation module 1250 is also used to input the input content, reference information, content guidance information and the selected first generated video into the backbone network for iterative noise reduction processing to obtain the second generated video: add noise based on the selected first generated video to obtain a noisy image sequence; input the noisy image sequence, the encoded reference image, the encoded detail information, the similarity information, the content guidance information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a second image feature sequence; decode the second image feature sequence through the encoding module to obtain the second generated video.
[0213] For more information about each module, see Figure 2-Figure 10 The relevant description of will not be repeated here. It should be understood that Fig.11 and Fig.12 The device and its modules shown can be implemented in various ways. For example, in some embodiments, the device and its modules can be implemented by hardware, software, or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or control codes contained in a processor, such as a disk, CD or DVD-ROM, etc., and such codes are provided in a carrier medium, or a memory of a programmable device. The system and its modules of this specification can not only be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software such as executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (e.g., firmware).
[0214] It should be noted that the above description of the system and its modules is only for convenience of description and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules without deviating from this principle to form a subsystem connected to other modules. Or some modules may be split to obtain more modules or multiple units under the module. Such variations are all within the scope of the disclosure of this specification.
[0215] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are taught in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.
Claims
1. A training method for a video generation model, characterized in that: include: Acquire a training sample, wherein the training sample includes a continuous picture sequence and a text description suitable for describing the picture sequence; Determining reference information according to the picture sequence, the reference information including a reference picture related to the picture sequence and similarity information between the reference picture and the picture sequence; Iteratively train the video generation model according to the training samples and the reference information to obtain a first-stage video generation model and a corresponding first model weight; The first model weight is used as the initial weight, and the video generation model obtained in the first stage is iteratively trained according to the training samples and the reference information to obtain the video generation model of the second stage and the corresponding second model weight, wherein the resolution of the picture sequence and the reference image in the second stage is greater than the resolution of the picture sequence and the reference image in the first stage.
2. The method according to claim 1, characterized in that The reference information also includes detail information extracted from the reference image.
3. The method according to claim 2, characterized in that Determining reference information according to the picture sequence includes: Determine any one picture in the picture sequence as a reference picture; extracting high frequency details in the reference image; The similarity coefficient between the reference image and each image in the image sequence is calculated according to the structural similarity.
4. The method according to claim 1 or 2, characterized in that: Also includes: Preprocessing the picture sequence so that the image resolution of the picture sequence in the second stage is greater than the image resolution of the picture sequence in the first stage; The reference information of the first stage and the second stage are determined based on the preprocessed picture sequence.
5. The method according to claim 2, characterized in that: The video generation model includes an encoding module and a backbone network including a timing module, the encoding module is suitable for encoding the training samples, the reference images and the detail information, the backbone network is suitable for predicting the noise distribution according to the input noisy image based on the diffusion principle, and the timing module is suitable for capturing the temporal correlation between images.
6. The method according to claim 5, characterized in that Also includes: The similarity information is determined according to the encoded reference image and the encoded picture sequence.
7. The method according to claim 5, characterized in that Iteratively training the video generation model according to the training sample and the reference information includes: Perform noise processing on the encoded image sequence to obtain a noisy image and input noise; Inputting the noisy image, the similarity information, the encoded reference image, the encoded detail information, the encoded text description and a preset timestamp into the backbone network; Obtaining output noise output by the backbone network corresponding to a preset timestamp; Adjusting the weight of the first-stage video generation model according to the input noise and the output noise; Repeat the above process for iterative training.
8. The method according to claim 7, characterized in that Adjusting the weight of the video generation model according to the input noise and the output noise includes: Determining loss parameter information according to the input noise and the output noise; The weight of the video generation model is adjusted according to the loss parameter information.
9. The method according to claim 8, characterized in that The method further comprises: Freezing the pre-trained weights of the encoding module and the pre-trained weights of the preset input layer in the backbone network; The weights of the unfrozen input layer in the backbone network and the weights of the timing module are adjusted according to the loss parameter information.
10. The method according to claim 5, characterized in that The video generation model further includes a content guidance module, and the method further includes: Determining content guidance information by a content guidance module, wherein the content guidance information is determined based on a reference image and a text description; The first model weight is used as the initial weight, and the video generation model obtained in the first stage is iteratively trained according to the training samples, reference information and the content guidance information to obtain the video generation model of the second stage and the corresponding second model weight.
11. The method according to claim 10, characterized in that The content guidance information determined by the content guidance module includes: Extracting picture feature information from the reference picture by a content guiding module, wherein the picture feature information is suitable for expressing the content presented in the reference picture; Mapping the image feature information to the same dimension as the encoded text description; The image feature information of the same dimension is concatenated with the text description to obtain content guidance information.
12. The method according to claim 10, characterized in that Iteratively training the video generation model obtained in the first stage according to the training samples, the reference information and the content guidance information includes: Perform noise processing on the encoded image sequence to obtain a noisy image and input noise; Inputting the noisy image, the similarity information, the encoded reference image, the encoded detail information, the content guide information and a preset timestamp into the backbone network; Obtaining output noise output by the backbone network corresponding to a preset timestamp; Adjusting the weight of the second-stage video generation model according to the input noise and the output noise; Repeat the above process for iterative training.
13. A video generation method, characterized in that: include: Acquire input content, the input content including an input reference image and a text description corresponding to the reference image; Determining reference information according to the reference graph, wherein the reference information includes similarity information related to the input content; Inputting the input content and the reference information into a pre-trained first-stage video generation model to obtain a first generated video; Get the selected first generated video; The input content, the reference information and the selected first generated video are input into a pre-trained second-stage video generation model to obtain a second generated video, wherein the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
14. The method according to claim 13, characterized in that The reference information also includes detail information extracted from the reference image.
15. The method according to claim 14, characterized in that Determining reference information according to the input content includes: extracting high frequency details in the reference image; A similarity coefficient related to the input content is generated through a preset periodic function.
16. The method according to claim 13 or 14, characterized in that Also includes: Preprocessing the reference image so that the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model; The reference information input into the first-stage video generation model and the second-stage video generation model are determined based on the preprocessed reference image.
17. The method according to claim 13, characterized in that Before obtaining the selected first generated video, it also includes: Change the random seed and generate two more first generation videos.
18. The method according to claim 14, characterized in that The video generation model includes an encoding module and a backbone network including a timing module. The encoding module is suitable for encoding the input content and the detail information and decoding the image features output by the backbone network. The backbone network is suitable for iteratively denoising the input noisy image based on the diffusion principle and outputting image features. The timing module is suitable for capturing the temporal correlation between images.
19. The method according to claim 18, characterized in that Inputting the input content and the reference information into a pre-trained first-stage video generation model to obtain a first generated video includes: Obtaining two or more preset noises consistent with the dimensions of the encoded reference image; Inputting two or more preset noises, the encoded reference image, the encoded detail information, the encoded text description, the similarity information and a preset timestamp into the backbone network for iterative noise reduction processing to obtain a first image sequence latent vector; The first image sequence latent vector is decoded by an encoding module to obtain a first generated video.
20. The method according to claim 14, characterized in that The second-stage video generation model also includes a content guidance module, and the method also includes: Determining content guidance information by a content guidance module, wherein the content guidance information is determined based on a reference image and a text description; The input content, the reference information, the content guide information and the selected first generated video are input into the backbone network for iterative noise reduction processing to obtain a second generated video.
21. The method according to claim 20, characterized in that The content guidance information determined by the content guidance module includes: Extracting picture feature information from the reference picture by a content guiding module, wherein the picture feature information is suitable for expressing the content presented in the reference picture; Mapping the image feature information to the same dimension as the encoded text description; The image feature information of the same dimension is concatenated with the text description to obtain content guidance information.
22. The method according to claim 19, characterized in that Inputting the input content, the reference information, the content guide information, and the selected first generated video into the backbone network for iterative noise reduction processing to obtain a second generated video includes: Adding noise based on the selected first generated video to obtain a noisy image sequence; Inputting the noisy image sequence, the encoded reference image, the encoded detail information, the similarity information, the content guide information and the preset timestamp into the backbone network for iterative noise reduction processing to obtain a second image feature sequence; The second image feature sequence is decoded by an encoding module to obtain a second generated video.
23. A training device for a video generation model, characterized in that: include: An acquisition module, used for acquiring a training sample, wherein the training sample includes a continuous picture sequence and a text description suitable for describing the picture sequence; A reference information determination module, configured to determine reference information according to the picture sequence, wherein the reference information includes a reference image related to the picture sequence and similarity information between the reference image and the picture sequence; A first-stage training module, used for iteratively training the video generation model according to the training samples and the reference information to obtain a first-stage video generation model and a corresponding first model weight; The second stage training module is used to use the first model weight as the initial weight, and iteratively train the video generation model obtained in the first stage according to the training samples and the reference information to obtain the second stage video generation model and the corresponding second model weight, wherein the resolution of the picture sequence and the reference picture in the second stage is greater than the resolution of the picture sequence and the reference picture in the first stage.
24. A video generating device, characterized in that: include: A content acquisition module, used to acquire input content, wherein the input content includes an input reference image and a text description corresponding to the reference image; A reference information determination module, configured to determine reference information according to the reference graph, wherein the reference information includes similarity information related to the input content; A first video generation module, used for inputting the input content and the reference information into a pre-trained first-stage video generation model to obtain a first generated video; A video acquisition module, used to acquire the selected first generated video; The second video generation module is used to input the input content, the reference information and the selected first generated video into a pre-trained second-stage video generation model to obtain a second generated video, wherein the resolution of the reference image input into the second-stage video generation model is greater than the resolution of the reference image input into the first-stage video generation model.
25. An electronic device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training method of the video generation model as described in any one of claims 1 to 12, or to implement the video generation method as described in any one of claims 13 to 22.
26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the video generation model as described in any one of claims 1 to 12, or to implement the video generation method as described in any one of claims 13 to 22.
Citation Information
Patent Citations
Video generation method and device and electronic equipment
CN116012479A
Video generation method and device, model training method and device, equipment and medium
CN116320216A
Content generation method and device based on text cue word and image driving and medium
CN117911584A
Video generation method and video analysis model training system
CN118053551A
Video synthesis via multimodal conditioning
US20230262293A1
