Model training method and device, video generation method and device and storage medium

Abstract

The invention relates to a model training method and device, a video generation method and device and a storage medium, and the method comprises the steps: obtaining training data; wherein the training data comprises video data and image data; training based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model for generating an image by using a text; the video generation model is a diffusion model for generating a video by using an image. According to the method and the device, the video data and the image data, which are single-mode data, are taken as the training data, so that training can be completed without collecting and constructing video text pairs, and the cost of training data construction and model training is reduced. And meanwhile, training is performed on the basis of the pre-trained image diffusion model to obtain a high-quality video generation model, so that the training efficiency and the training effect are effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a model training method, a video generation method, a device and a storage medium. Background Art

[0002] AI-generated content (AIGC) refers to content generated by generative AI technology. Image generation based on diffusion models is a specific application scenario of AIGC, such as diffusion models such as Midjourney and Stable Diffusion. With the help of controllable generation technologies such as Controlnet, fine-tuning technologies such as Dreambooth and Lora, and image editing technologies such as Text Inversion, users can input text into the diffusion model as needed to generate the desired image. Compared with image generation based on diffusion models, video generation based on diffusion models is more difficult and challenging; in the prior art, text or text plus additional control information is input into the diffusion model to generate videos. This diffusion model is difficult to train and the content of the generated video is poorly controllable. Summary of the invention

[0003] In view of this, the present disclosure proposes a model training method, a video generation method, a device, a storage medium and a computer program product.

[0004] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0005] Acquire training data; wherein the training data includes video data and image data;

[0006] Performing training based on the training data and a pre-trained image diffusion model to obtain a video generation model;

[0007] Among them, the image diffusion model is a diffusion model that generates images from texts; and the video generation model is a diffusion model that generates videos from images.

[0008] In a possible implementation, the training based on the training data and the pre-trained image diffusion model to obtain a video generation model includes:

[0009] Adjusting the structure of the image diffusion model to obtain an image diffusion model with adjusted structure;

[0010] Using the image data to train the image diffusion model after the structure adjustment to obtain an intermediate diffusion model, wherein the intermediate diffusion model is a diffusion model for generating images from images;

[0011] Adjusting the structure of the intermediate diffusion model to obtain an intermediate diffusion model with adjusted structure;

[0012] The video data is used to train the intermediate diffusion model after the structural adjustment to obtain the video generation model.

[0013] In a possible implementation, the image diffusion model includes: a text encoder, a convolutional encoder-decoder structure, and a variational autoencoder;

[0014] The step of adjusting the structure of the image diffusion model to obtain the image diffusion model after the structure adjustment includes:

[0015] The text encoder is replaced by a visual encoder, and a linear mapping layer is configured after the visual encoder to obtain the image diffusion model after the structure adjustment;

[0016] The visual encoder is used to extract image features, and the linear mapping layer is used to align the image features with the text features extracted by the text encoder.

[0017] In one possible implementation, aligning the image features with the text features extracted by the text encoder includes: flattening the input features of the global pooling layer in the visual encoder along the spatial dimension to obtain a visual feature sequence, and aligning the length of the visual feature sequence with the length of the text feature.

[0018] In a possible implementation manner, the using the image data to train the image diffusion model after the structure adjustment includes:

[0019] Inputting the image data into the structure-adjusted image diffusion model, extracting a first feature of the image data through the visual encoder, and extracting a second feature of the image data through the variational autoencoder;

[0020] Inputting the first feature into a linear mapping layer for alignment processing to obtain an aligned first feature;

[0021] Inputting the aligned first feature and the second feature into a convolutional encoder-decoder structure for processing to obtain a predicted image;

[0022] According to the predicted image and the image data, the parameters of the linear mapping layer and each transformer block in the convolutional encoder-decoder structure are adjusted.

[0023] In a possible implementation manner, adjusting the structure of the intermediate diffusion model to obtain the intermediate diffusion model after structural adjustment includes:

[0024] A timing module is added after each transformer block in the convolutional encoder-decoder structure, and a splicing module is added before the convolutional encoder-decoder structure to obtain the intermediate diffusion model after the structure adjustment, wherein the timing module is used to calculate the temporal self-attention to align the content of the first video frame in the video data with the content of each video frame generated by the convolutional encoder-decoder structure; and the splicing module is used to splice each video frame in the video data.

[0025] In a possible implementation manner, the using the video data to train the intermediate diffusion model after the structure adjustment includes:

[0026] Extracting a second feature of each video frame in the video data using the variational autoencoder;

[0027] Using the splicing module, the second feature of the first video frame in the video data is spliced ​​with the second features of other video frames after noise is added to obtain a spliced ​​feature;

[0028] Inputting a first video frame in the video data into the visual encoder, extracting a first feature of the first video frame in the video data, and aligning the first feature through a linear mapping layer;

[0029] Inputting the spliced ​​features and the first feature of the first video frame in the aligned video data into the convolutional encoder-decoder structure for denoising to generate a predicted video;

[0030] The parameters of the timing module are adjusted according to the predicted video and the video data.

[0031] In a possible implementation manner, adjusting the parameters of the timing module according to the predicted video and the video data includes:

[0032] Calculate the loss function value according to other video frames except the first video frame in the video data and other video frames except the first video frame in the predicted video;

[0033] Based on the loss function value, the parameters of the timing module are adjusted.

[0034] In a possible implementation manner, the using the video data to train the intermediate diffusion model after the structure adjustment includes:

[0035] Performing dimensionality reduction processing on the video data to obtain dimensionality-reduced data; wherein the dimensionality of the dimensionality-reduced data is the same as the dimensionality of the image data;

[0036] The intermediate diffusion model after structural adjustment is trained using the dimension-reduced data.

[0037] In a possible implementation manner, an initial value of a parameter of the timing module is zero.

[0038] According to another aspect of the present disclosure, a video generation method is provided, the method comprising:

[0039] Get a reference image;

[0040] The reference image is processed using a target model to generate a target video; wherein the target model is a video generation model obtained by any of the above-mentioned model training methods.

[0041] In a possible implementation, the target video includes a preset number of video frames;

[0042] The method further comprises:

[0043] Extract the last video frame in the target video;

[0044] Using the last video frame as the reference image;

[0045] Repeat the process of processing the reference image using the target model to generate a target video, and subsequent operations until the number of video frames in the stitched video reaches a preset threshold, wherein the stitched video is obtained by stitching together the target videos generated by processing each reference image with the target model.

[0046] In a possible implementation, the using the target model to process the reference image to generate the target video includes:

[0047] Extracting a first feature of the reference image using a visual encoder in the target model, and aligning the first feature through a linear mapping layer in the target model to obtain an aligned first feature;

[0048] Extracting a second feature of the reference image using a variational autoencoder in the target model, and concatenating the second feature with the random noise to obtain a concatenated feature;

[0049] The aligned first feature and the spliced ​​feature are input into the convolutional encoder-decoder structure in the target model for processing to generate the target video.

[0050] According to another aspect of the present disclosure, a model training device is provided, the device comprising:

[0051] An acquisition module, used to acquire training data; wherein the training data includes video data and image data;

[0052] A training module is used to perform training based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model that generates images from texts; and the video generation model is a diffusion model that generates videos from images.

[0053] According to another aspect of the present disclosure, a video generating device is provided, the device comprising:

[0054] An acquisition module, used for acquiring a reference image;

[0055] A generation module is used to process the reference image using a target model to generate a target video; wherein the target model is a video generation model obtained by any of the above-mentioned model training methods.

[0056] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0057] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0058] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0059] Through various aspects of the present disclosure, training data is obtained; training is performed based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model for generating images from text; and the video generation model is a diffusion model for generating videos from images; wherein the training data includes video data and image data, so that the single-modality data of video data and image data is used as training data, and training can be completed without collecting and constructing video-text pairs, thereby reducing the cost of training data construction and model training. At the same time, training is performed on the basis of a pre-trained image diffusion model to obtain a high-quality video generation model, thereby effectively improving training efficiency and training effect.

[0060] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0062] Figure 1 A flow chart showing a model training method according to an embodiment of the present disclosure is shown;

[0063] Figure 2 A flow chart showing a model training method according to an embodiment of the present disclosure is shown;

[0064] Figure 3 A schematic diagram showing the structure of a timing module according to an embodiment of the present disclosure is shown;

[0065] Figure 4 A schematic diagram showing a method of training an intermediate diffusion model after structural adjustment according to an embodiment of the present disclosure is shown;

[0066] Figure 5 A flowchart of a video generation method according to an embodiment of the present disclosure is shown;

[0067] Figure 6 A schematic diagram showing a method of generating a target video according to an embodiment of the present disclosure is shown;

[0068] Figure 7 A flowchart of a video generation method according to an embodiment of the present disclosure is shown;

[0069] Figure 8 A structural diagram of a model training device according to an embodiment of the present disclosure is shown;

[0070] Fig. 9 A structural diagram of a video generating device according to an embodiment of the present disclosure is shown;

[0071] Fig.10 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0073] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present disclosure include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the phrases "exemplary", "in one embodiment", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0074] In the present disclosure, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or plural.

[0075] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0076] AI-generated content (AIGC) refers to content generated by generative AI technology. Image generation based on the diffusion model is a specific application scenario of AIGC. Users can input text into the diffusion model according to their needs and desires to generate the desired pictures. Compared with image generation based on the diffusion model, video generation based on the diffusion model is more difficult and challenging. According to the interactive mode of generating videos, the video generation technology based on the diffusion model in related technologies can be mainly divided into two directions:

[0077] (1) Video generation completely controlled by text, that is, the content of the generated video is controlled by text prompts. This type of technology usually adds some modules for temporal modeling, such as one-dimensional convolutional layers and self-attention layers, based on the two-dimensional image diffusion model. Since the parameters of the two-dimensional image diffusion model are reused, this type of method usually only requires a small amount of fine-tuning training or no training to achieve the generation from text to video. However, the consistency between the video frames in the videos generated by this method is poor, with many jitters and incoherence; using a large number of text-video pairs for training can improve this problem, but since text-video pairs are very scarce, it is still very difficult to train a high-quality text-to-video generation model.

[0078] (2) Generate video by adding additional control information to text. Commonly used control information includes optical flow, depth map, key points, etc. This control information can be extracted from the template video uploaded by the user or generated from text. Due to the introduction of additional control information, the video images generated by this type of technology are smoother and more stable. However, this additional control information usually only provides structural information for the generated video. The semantic content of the generated video is still determined by the text. Therefore, complex text prompt word engineering is still required to produce the desired picture content. The user interactivity is poor and the controllability of the generated video content is poor.

[0079] In order to solve the above technical problems, the embodiments of the present disclosure propose a model training method and a video generation method (for detailed description, see below). In the training stage, training is performed based on video data, image data and a pre-trained image diffusion model to obtain a video generation model, wherein the image diffusion model is a diffusion model for generating images from text; the video generation model is a diffusion model for generating videos from images; thus, the training can be completed without collecting and constructing video-text pairs, thereby reducing the cost of training data construction and model training. In the inference stage, based on the video generation model, the user does not need to input text prompt words, but only needs to upload a picture to generate a video that is semantically consistent with the picture uploaded by the user and is smooth, which has the characteristics of good versatility, user-friendly interaction, and high controllability of video content.

[0080] Exemplarily, the model training method and the video generation method can be executed by a processor of an electronic device. The electronic device can be a terminal device, a server or a virtual machine, or a distributed computer system composed of one or more servers and / or computers. The terminal device includes but is not limited to: a smart phone, a laptop computer, a desktop computer, a platform computer, a vehicle-mounted device, a smart wearable device, etc. The server can be an ordinary server or a cloud server, which is also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0081] Figure 1 A flow chart of a model training method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method may include the following steps:

[0082] Step 101: Acquire training data; wherein the training data includes video data and image data.

[0083] As an example, the video data includes a preset number of video frames. Exemplarily, the original video data can be selected from a large-scale public video data set; and then a preset number of continuous video frames are sampled from the original video data at a fixed frame rate (fps) as the video data, wherein the fixed frame rate and the preset number have the same value and can be set as needed, for example, 8, 16, etc.

[0084] Exemplarily, the image data can be obtained by extracting frames from the above-mentioned video data. For example, the first frame in the video data can be used as the image data. The image data can also be selected from some high-quality open source image datasets used to train generative models, such as laion-aesthetic, laion-5B, etc.

[0085] Exemplarily, the training data may include multiple batches of video data and multiple image data.

[0086] Step 102: Perform training based on the training data and a pre-trained image diffusion model to obtain a video generation model.

[0087] The image diffusion model is a diffusion model that generates images from text, which can also be called a two-dimensional image diffusion model; that is, the image diffusion model is a diffusion model that uses text prompt words as input data and outputs images. The video generation model is a diffusion model that generates videos from images, that is, the video generation model is a diffusion model that uses images as input data and outputs videos.

[0088] Among them, the diffusion model can also be called the diffusion generation model, which is a generation model used to generate data, such as images, text, videos, etc. The core idea of ​​the diffusion model is to add noise to the data and then gradually remove the noise to restore the original data. Since the image diffusion model is a pre-trained diffusion model, the text features can be used as a conditional input and trained as a priori information through the cross-attention mechanism, and it has the ability to extract features clearly and accurately; at the same time, the video data and image data are large in size and usually of high quality. In this way, using the pre-trained image diffusion model as the base film, using video data and image data as training data, a high-quality video generation model is trained, which effectively improves the training efficiency and training effect.

[0089] It should be noted that the pre-trained image diffusion model may have no restrictions on the model structure, that is, it can be trained based on an image diffusion model of any structure to obtain a video generation model, thereby improving the versatility of the model training method.

[0090] In the disclosed embodiment, training data is obtained; training is performed based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model that generates images from text; and the video generation model is a diffusion model that generates videos from images; wherein the training data includes video data and image data. In this way, the single-modality data of video data and image data is used as training data, and training can be completed without collecting and constructing video-text pairs, thereby reducing the cost of training data construction and model training. At the same time, training is performed on the basis of the pre-trained image diffusion model to obtain a high-quality video generation model, thereby effectively improving training efficiency and training results.

[0091] Furthermore, in the above step 102, the training based on the training data and the pre-trained image diffusion model can be divided into two training stages. Through the training of these two stages, a high-quality video generation model can be obtained.

[0092] Figure 2 A flow chart of a model training method according to an embodiment of the present disclosure is shown. Figure 2 As shown, the method may include the following steps:

[0093] Step 201: Adjust the structure of the image diffusion model to obtain an image diffusion model after structural adjustment.

[0094] In this step, the structure of the pre-trained image diffusion model is adjusted so as to process image data using the pre-trained image diffusion model.

[0095] As an example, the image diffusion model includes: a text encoder, a convolutional encoder-decoder structure (UNet), and a variational auto-encoder (VAE); in step 201, the structure of the image diffusion model is adjusted to obtain the image diffusion model after the structure adjustment, including: replacing the text encoder with a visual encoder, and configuring a linear mapping layer after the visual encoder to obtain the image diffusion model after the structure adjustment; wherein the visual encoder is used to extract image features (also referred to as image embedding coding), and the linear mapping layer is used to align the image features with the text features extracted by the text encoder (also referred to as text embedding coding). Exemplarily, the variational auto-encoder is used to extract VAE features of input data, and the VAE features are a representation of a latent space. As an effective starting point for the diffusion model to work, the diffusion model can perform denoising and denoising based on the VAE features to generate a prediction result.

[0096] As an example, the text encoder may be a text encoder in a Contrastive Language-Image Pre-Training (Clip) model; and the visual encoder may be a visual encoder in a Clip model. The length of the text feature extracted by the text encoder is known and may be a parameter in the text encoder.

[0097] Correspondingly, the structurally adjusted image diffusion model includes a visual encoder, a convolutional encoder-decoder structure, a variational autoencoder, and a linear mapping layer. Since the visual encoder replaces the text encoder in the image diffusion model before adjustment (i.e., the pre-trained image diffusion model), and the text and the image are different modal data, the image features extracted by the visual encoder differ in length from the text features extracted by the original text encoder. Then, the configured linear mapping layer is used to process the image features extracted by the visual encoder so that the processed image features are aligned in length with the text features extracted by the original text encoder. In this way, the aligned image features have the same length as the text features originally input into the convolutional encoder-decoder structure, so that the convolutional encoder-decoder structure can process the aligned image features.

[0098] Exemplarily, the alignment of the image features with the text features extracted by the text encoder includes: flattening the input features of the global pooling layer in the visual encoder along the spatial dimension to obtain a visual feature sequence, and aligning the length of the visual feature sequence with the length of the text feature. In this way, the input features of the last global pooling layer in the visual encoder are flattened along the spatial dimension so that more detailed image features can be retained. The size of the input features of the global pooling layer can be expressed as H*W, where H represents the height dimension, that is, the number of vertical tokens; W represents the width dimension, that is, the number of horizontal tokens; all tokens of the input features of the global pooling layer are arranged in order into a one-dimensional visual feature sequence, which includes H*W tokens, thereby achieving flattening along the spatial dimension; after these H*W tokens pass through the linear mapping layer, the length is aligned with the length of the text feature.

[0099] Step 202: Use the image data to train the image diffusion model after the structure adjustment to obtain an intermediate diffusion model, where the intermediate diffusion model is a diffusion model that generates images using images.

[0100] In this step, the image diffusion model after the above-mentioned structural adjustment is trained. Exemplarily, during the training process, all parameters of the visual encoder are frozen, and only the parameters of the linear mapping layer are adjusted; at the same time, for the convolutional encoder-decoder structure, all parameters of other components in the convolutional encoder-decoder structure except the transformer block are frozen, and only the parameters of each transformer block in the convolutional encoder-decoder structure are fine-tuned.

[0101] In a possible implementation, in step 202, the use of the image data to train the structurally adjusted image diffusion model includes: inputting the image data into the structurally adjusted image diffusion model, extracting a first feature of the image data through the visual encoder, and extracting a second feature of the image data (i.e., a VAE feature) through a variational autoencoder; inputting the first feature into a linear mapping layer for alignment processing to obtain an aligned first feature; inputting the aligned first feature and the second feature into a convolutional encoder-decoder structure for processing to obtain a predicted image; and adjusting the parameters of each transformer block in the linear mapping layer and the convolutional encoder-decoder structure according to the predicted image and the image data.

[0102] Exemplarily, the loss function value can be calculated according to the gap between the predicted image and the image data, and back propagation can be performed according to the loss function value to adjust the parameters of the linear mapping layer and each transformer block in the convolutional encoder-decoder structure until the training termination condition is met, wherein the training termination condition may be that the loss function value is less than a specific value, the training time reaches a specific value, the number of iterations reaches a specific value, etc., thereby obtaining a trained intermediate diffusion model.

[0103] In this way, through the above steps 201 and 202, the first stage of training is completed; after the first stage of training, a high-quality intermediate diffusion model is obtained, and the intermediate diffusion model can use an image as input data to generate an image similar to the image. The first stage of training can also be called the image variance stage; then based on the intermediate diffusion model, the second stage of training is performed.

[0104] Step 203: Adjust the structure of the intermediate diffusion model to obtain an intermediate diffusion model after structural adjustment.

[0105] Exemplarily, the intermediate diffusion model may include: a visual encoder, a convolutional encoder-decoder structure, a variational autoencoder, and a linear mapping layer.

[0106] Exemplarily, in the second stage of training, the parameters of the intermediate diffusion model trained in the first stage are reused, and a timing module is added to the structure of the intermediate diffusion model to obtain an intermediate diffusion model with adjusted structure.

[0107] In a possible implementation, the step 203 of adjusting the structure of the intermediate diffusion model to obtain the intermediate diffusion model after structural adjustment may include: adding a timing module after each transformer block in the convolutional encoder-decoder structure, and adding a splicing module before the convolutional encoder-decoder structure to obtain the intermediate diffusion model after structural adjustment, wherein the timing module is used to calculate the temporal self-attention to align the first video frame in the video data with the content of each video frame generated by the convolutional encoder-decoder structure; the splicing module is used to splice each video frame in the video data. Accordingly, the intermediate diffusion model after structural adjustment includes a visual encoder, a convolutional encoder-decoder structure, a variational autoencoder, a linear mapping layer, and a splicing module; wherein the convolutional encoder-decoder structure includes a timing module.

[0108] Figure 3 A schematic diagram of the structure of a timing module according to an embodiment of the present disclosure is shown; Figure 3As shown in the figure, the yellow one is the deep residual network block (ResNet block) in the convolutional encoder-decoder structure, and the red one is the transformer block (transformer block) in the convolutional encoder-decoder structure. The timing module (motion module) is configured after the transformer block. The transformer block extracts the intermediate features corresponding to the video data and outputs them to the timing module. The timing module calculates the temporal self-attention and extracts the spatiotemporal features. That is, for each point in the spatial dimension of the video feature, the self-attention will be calculated along the time dimension T, so as to align the content of the first video frame in the video generated by the convolutional encoder-decoder structure with that of other video frames; Figure 3 The blue square in the middle represents a point in the spatial dimension. The timing module calculates the self-attention (1D-Attention) of this point along the time dimension T, so that the features of this point in adjacent video frames in the generated video are aligned in time.

[0109] Step 204: Use the video data to train the intermediate diffusion model after the structural adjustment to obtain the video generation model.

[0110] Exemplarily, the first frame in the video data is used as a reference frame to guide the intermediate diffusion model after structural adjustment to generate a video; and other frames in the video data are used as training target frames to compare with corresponding frames in the video generated by the intermediate diffusion model.

[0111] The training of the second stage reuses the parameters of the intermediate diffusion model trained in the first stage. As an example, the intermediate diffusion model after structural adjustment is obtained by adding a timing module to the structure of the intermediate diffusion model trained in the first stage. Since the input data of the intermediate diffusion model trained in the first stage is image data, and the training of the second stage uses video data as input data; therefore, it is necessary to perform an "expansion" operation to reorganize the shape of the video data and reduce the dimension of the video data so that the intermediate diffusion model after structural adjustment can process video data.

[0112] In a possible implementation, the using the video data to train the intermediate diffusion model after structural adjustment includes: performing dimensionality reduction processing on the video data to obtain dimension-reduced data; wherein the dimension of the dimension-reduced data is the same as the dimension of the image data; and using the dimension-reduced data to train the intermediate diffusion model after structural adjustment.

[0113] Exemplarily, the dimension of the video data includes at least one of a time dimension, a batch dimension, a height dimension, a width dimension, and a channel dimension.

[0114] For video data, since it contains multiple consecutive video frames, the model can regard it as a multi-dimensional video tensor (Tensor); the multi-dimensionality may include: Batch Size: represents the number of videos processed simultaneously. During training, multiple samples are usually used for batch processing to improve computational efficiency; Time: represents the length of the video data, that is, the number of video frames; Height: represents the height of the video frame, that is, the number of vertical pixels; Width: represents the width of the video frame, that is, the number of horizontal pixels; Channel: represents the RGB color channel of the video frame. Exemplarily, the time dimension and batch dimension in the multidimensional video tensor can be put together to obtain a tensor with a shape of (BxT)xHxWxC, where B is the batch dimension, T is the time dimension, W is the width dimension, H is the height dimension, and C is the channel dimension; in this way, the shape of the multidimensional video tensor is reorganized, and the dimensionality reduction of the multidimensional video tensor is achieved, so that the video data can be used as the input data of the model without modifying the original network parameters and structure of the intermediate diffusion model after the structural adjustment, so that the model can process the video data.

[0115] In a possible implementation, the use of the video data to train the structurally adjusted intermediate diffusion model includes: using the variational autoencoder to extract the second feature of each video frame in the video data; using a splicing module to splice the second feature of the first video frame in the video data with the second features of other video frames after noise addition to obtain a spliced ​​feature; inputting the first video frame in the video data into the visual encoder, extracting the first feature of the first video frame in the video data, and aligning them through a linear mapping layer; inputting the spliced ​​feature and the aligned first feature of the first video frame in the video data into the convolutional encoder-decoder structure for denoising to generate a predicted video; and adjusting the parameters of the timing module according to the predicted video and the video data. In this way, the structurally adjusted intermediate diffusion model is used as the base model, the first video frame in the video data is used as the first frame, and the first feature of the first video frame in the aligned video data and the second feature of the first video frame in the spliced ​​video data are used as the second feature of other video frames after noise addition to obtain the spliced ​​feature, which is input into the convolutional encoder-decoder structure, so that the convolutional encoder-decoder structure generates subsequent other video frames in the predicted video based on the picture content of the first frame to obtain subsequent motion changes; when adjusting the parameters of the model, the parameters of other components in the structurally adjusted intermediate diffusion model are fixed, and the parameters of the timing module are fine-tuned to achieve a more consistent alignment effect between the video frames in the generated video.

[0116] Exemplarily, adjusting the parameters of the timing module according to the predicted video and the video data includes: calculating the loss function value according to other video frames in the video data except the first video frame, and other video frames in the predicted video except the first video frame; adjusting the parameters of the timing module based on the loss function value. Since the second feature of the first video frame is not denoised, only other video frames need to be considered when calculating the loss function value; exemplarily, back propagation can be performed according to the loss function value to adjust the parameters of the timing module until the training termination condition is met, wherein the training termination condition can be that the loss function value is less than a specific value, the training time reaches a specific value, the number of iterations reaches a specific value, etc., so as to obtain a trained video generation model.

[0117] Exemplarily, the initial value of the parameters of the timing module is zero. That is, the parameter values ​​of the timing module are adjusted from zero during the training process. In this way, the parameters of the newly added timing module are initialized by zero initialization, and the bottom model parameters other than the timing module in the intermediate diffusion model after structural adjustment are frozen. Through fine-tuning, it is ensured that the inserted timing module does not destroy the feature distribution learned by the original intermediate diffusion model.

[0118] For example, Figure 4 A schematic diagram of training a structure-adjusted intermediate diffusion model according to an embodiment of the present disclosure is shown, as shown in FIG. Figure 4As shown in the figure, the intermediate diffusion model after structural adjustment includes: the visual encoder of the CLIP model, the linear mapping layer, the convolutional encoder-decoder structure (UNet), the variational autoencoder (VAE), and the splicing module; among them, UNet includes multiple residual network blocks (ResNet block), transformer blocks (transformerblock) and timing modules (motion module). In the figure, the yellow one is the deep residual network block, the red one is the transformer block, and the blue one is the timing module. In order to strengthen the content consistency between each frame in the video generated by the model and the first frame of the video, the first frame in the video data is used as a reference image, where the video data can be represented as T*3*H*W, that is, the video data contains T video frames, each video frame has 3 color channels, and the size of each video frame is H*W. The VAE features of each video frame in the video data are extracted by the variational autoencoder; the VAE features of the first video frame in the video data do not add noise, but the VAE features of the first video frame in the video data are spliced ​​with the VAE features of other video frames after noise addition and sent to UNet, where the added noise is (T-1)*3*H*W. At the same time, the features of the first video frame in the video data extracted by the visual encoder are aligned and input into UNet, and UNet performs denoising to generate a predicted video, and the video frames in the predicted video are based on the picture content of the first video frame to obtain subsequent motion changes.

[0119] In this way, the second stage of training is completed in the above steps 203 and 204. The video generation model obtained by training can generate smoother video animation.

[0120] In the disclosed embodiment, a video generation model guided by a reference image is trained through a two-stage training strategy, which strengthens the consistency and coherence of the video content generated by the video generation model with the reference image. In the first stage of training, the structure of the pre-trained image diffusion model is adjusted to replace the embedded coding of the text in the original image diffusion model with the embedded coding of the image, and the image diffusion model after the structure adjustment is trained using the image data to obtain an intermediate diffusion model, which can generate pictures similar to the image data. In the second stage of training, the structure of the intermediate diffusion model obtained in the first stage is adjusted, and the intermediate diffusion model after the structure adjustment is continued to be trained using the video data; for example, a timing module can be inserted into the intermediate diffusion model obtained in the first stage to align the subsequent video frames generated by the model with the content of the first frame through the timing module, so as to guide the model to generate coherent and consistent video content through the first frame of the video data. In this way, the trained video generation model can generate high-quality, smooth, coherent and consistent videos by inputting only one reference image.

[0121] Figure 5 FIG. 1 is a flow chart of a video generation method according to an embodiment of the present disclosure. Figure 5 As shown, the method may include the following steps:

[0122] Step 501: Obtain a reference image.

[0123] Exemplarily, a reference image may be obtained.

[0124] The reference image is used to control the picture content of the generated video.

[0125] Step 502: Process the reference image using the target model to generate a target video; wherein the target model is obtained by Figure 1 or Figure 2 The video generation model is obtained by training the model training method shown.

[0126] In this way, the reference image provided by the user is processed using the video generation model, and the video generation model generates a video under the guidance of the reference image.

[0127] Exemplarily, the target video includes a preset number of video frames, wherein the preset number is the same as the number of video frames in the video data used when training the target model. Exemplarily, the first frame in the target video is the reference image.

[0128] In a possible implementation, in step 502, the use of the target model to process the reference image to generate the target video may include: using the visual encoder in the target model to extract the first feature of the reference image, and aligning it through the linear mapping layer in the target model to obtain the aligned first feature; using the variational autoencoder in the target model to extract the second feature of the reference image (i.e., AVE feature), and splicing it with the random noise to obtain the splicing feature; inputting the aligned first feature and the splicing feature into the convolutional encoder-decoder structure in the target model for processing to generate the target video. In this way, the reference image provided by the user is used as the first frame of the target video, and the first feature and the splicing feature of the aligned reference image are input into the convolutional encoder-decoder in the target model, so that the convolutional encoder-decoder structure generates subsequent other video frames in the target video based on the picture content of the reference image provided by the user, and obtains subsequent motion changes, so that the alignment effect between the video frames in the generated target video is more consistent.

[0129] Exemplarily, extracting the second feature of the reference image by using the variational autoencoder in the target model and splicing it with random noise to obtain a spliced ​​feature can include: performing random noise addition on the second feature of the reference image extracted by the variational autoencoder for multiple times, wherein the number of random noise additions is the number of video frames in the target video minus one; and using the second feature after each random noise addition as the second feature corresponding to each video frame in the target video except the first frame; using the second feature of the reference image as the second feature corresponding to the first frame in the target video; and splicing the second feature corresponding to the first frame in the target video and the second feature corresponding to each video frame in the target video except the first frame to obtain a spliced ​​feature.

[0130] For example, Figure 6 A schematic diagram of generating a target video according to an embodiment of the present disclosure is shown as follows: Figure 6 As shown in the figure, the target model includes: the visual encoder of the CLIP model, the linear mapping layer, the convolutional encoder-decoder structure (UNet), the variational autoencoder (VAE), and the splicing module; among them, UNet includes multiple residual network blocks (ResNet block), transformer blocks (transformer block) and timing modules (motion module). In the figure, the yellow one is the deep residual network block, the red one is the transformer block, and the blue one is the timing module. The reference image provided by the user is the first frame of the target video. The VAE features of the reference image are extracted by the variational autoencoder, and the VAE features are spliced ​​with the random noise to obtain the spliced ​​features, which are input into UNet, where the random noise Noise is (T-1)*3*H*W, where T represents the time dimension, and its value is the number of T video frames in the target video, and H*W is the size of the reference image. At the same time, the features of the reference image provided by the user extracted by the visual encoder are aligned and input into Unet. In this way, the picture content of the reference image is used as the basis for the picture motion changes in the subsequently generated video frames. Unet generates the target video through denoising processing with several steps, thereby enhancing the content consistency between each frame in the target video and the reference image.

[0131] In the disclosed embodiments, based on the video generation model, a high-quality, smooth, coherent, and consistent target video is generated from a reference image provided by a user; the controllability is stronger, and the user can control the picture content of the generated target video through the reference image, and can generate videos of different styles, that is, a video of a corresponding style can be generated according to the style of the reference image input by the user; in this way, the user can more easily obtain the desired video content through the reference image without the need to design complex text prompts, which reduces the user's usage cost and improves the user's efficiency, and has great application value in video production and other fields.

[0132] Taking into account the limitation of computing resources during the training phase of the video generation model, it is necessary to input a video data containing a preset number of video frames into the intermediate diffusion model after the structural adjustment of the second phase training, and generate a predicted video containing the preset number of video frames. In this way, the trained video generation model can only generate videos with the same number of frames as the video data used during training. In order to generate videos of arbitrarily long duration, in the embodiment of the present disclosure, without the need for additional training of the video generation model, in the inference phase, longer videos can be generated in an autoregressive manner, that is, videos with more video frames than those in the video data used during training can be generated.

[0133] Figure 7 FIG. 1 is a flow chart of a video generation method according to an embodiment of the present disclosure. Figure 7 As shown, the method may include the following steps:

[0134] Step 701: Obtain a reference image.

[0135] This step is similar to the above Figure 5 The same as step 501.

[0136] Step 702: Use the target model to process the reference image to generate a target video.

[0137] This step is similar to the above Figure 5 The same as step 502.

[0138] Step 703: extract the last video frame in the target video;

[0139] Step 704: using the last video frame as the reference image;

[0140] Step 705, repeatedly executing the process of processing the reference image using the target model to generate a target video, and subsequent operations, until the number of video frames in the stitched video reaches a preset threshold, wherein the stitched video is obtained by stitching together the target videos generated by processing each reference image by the target model.

[0141] For example, the preset threshold value can be set according to the needs of the user. The value of the preset threshold value is greater than the number of video frames in the video data used when training the target model.

[0142] Exemplarily, in this step, after each target video is generated, the target videos generated by processing each reference image with the target model are spliced ​​to obtain a spliced ​​video; wherein, during the splicing process, after removing the same video frames in the target video generated by processing each reference image with the target model, that is, after removing the first frame of the target video generated by processing each reference image with the target model (except the reference image provided by the user), they are arranged in the order of generation to obtain a spliced ​​video.

[0143] For example, when training the target data, the number of video frames in the video data is 16 frames. First, the reference image provided by the user is used as the first frame, and the target model is used to generate the subsequent 15 frames. In this way, a 16-frame video is obtained through the first reasoning; then the last frame of the video is used as the reference image for the next video generation, that is, the last frame of the video is input into the target model, and the target model is used to infer the subsequent 15 frames to obtain another 16-frame video; the repeated video frames in the two generated videos, that is, the first frame in the second generated video, are removed, and the two obtained videos are spliced ​​to obtain a 31-frame long video. This is repeated, and the target model only needs to perform multiple reasonings to obtain a longer video. In this way, the last frame of the video newly inferred by the target model is used as a new reference image, and the target model is used to continuously generate the picture content after the new reference image, thereby continuously increasing the length of the spliced ​​image until the number of video frames in the spliced ​​video reaches a preset threshold, thereby realizing the autoregressive generation of a long video.

[0144] In the disclosed embodiment, the last frame in the video most recently inferred by the target model is used as a new reference image, and the reference image is repeatedly processed using the target model to generate a target video; each target video generated is spliced ​​until the number of video frames in the spliced ​​video reaches a preset threshold. In this way, a long-term video generation scheme based on autoregressive prediction is implemented without the need for additional adjustments during training, so that the target model trained with training data with a fixed number of frames can generate longer videos to meet user needs.

[0145] Based on the same inventive concept of the above method embodiment, the embodiment of the present disclosure also provides a model training device, which can be used to execute the technical solution described in the above method embodiment. Figure 1 or Figure 2 The steps of the model training method shown.

[0146] Figure 8 A structural diagram of a model training device according to an embodiment of the present disclosure is shown as follows: Figure 8As shown, the model training device may include: a first acquisition module 801, used to acquire training data; wherein the training data includes video data and image data; a training module 802, used to perform training based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model that generates images from text; and the video generation model is a diffusion model that generates videos from images.

[0147] In a possible implementation, the training module 802 is further used to: adjust the structure of the image diffusion model to obtain an image diffusion model with adjusted structure; train the image diffusion model with adjusted structure using the image data to obtain an intermediate diffusion model, where the intermediate diffusion model is a diffusion model that generates images from images; adjust the structure of the intermediate diffusion model to obtain an intermediate diffusion model with adjusted structure; and train the intermediate diffusion model with adjusted structure using the video data to obtain the video generation model.

[0148] In one possible implementation, the image diffusion model includes: a text encoder, a convolutional encoder-decoder structure, and a variational autoencoder; the training module 802 is also used to: replace the text encoder with a visual encoder, and configure a linear mapping layer after the visual encoder to obtain the image diffusion model after the structure adjustment; wherein the visual encoder is used to extract image features, and the linear mapping layer is used to align the image features with the text features extracted by the text encoder.

[0149] In a possible implementation, the training module 802 is further used to: flatten the input features of the global pooling layer in the visual encoder along the spatial dimension to obtain a visual feature sequence, and align the length of the visual feature sequence with the length of the text feature.

[0150] In a possible implementation, the training module 802 is also used to: input the image data into the image diffusion model after the structural adjustment, extract the first feature of the image data through the visual encoder, and extract the second feature of the image data through the variational autoencoder; input the first feature into the linear mapping layer for alignment processing to obtain the aligned first feature; input the aligned first feature and the second feature into the convolutional encoder-decoder structure for processing to obtain a predicted image; and adjust the parameters of each transformer block in the linear mapping layer and the convolutional encoder-decoder structure according to the predicted image and the image data.

[0151] In a possible implementation, the training module 802 is also used to: add a timing module after each transformer block in the convolutional encoder-decoder structure, and add a splicing module before the convolutional encoder-decoder structure to obtain the intermediate diffusion model after the structure adjustment, wherein the timing module is used to calculate the temporal self-attention to align the content of the first video frame in the video data with the content of each video frame generated by the convolutional encoder-decoder structure; the splicing module is used to splice each video frame in the video data.

[0152] In one possible implementation, the training module 802 is also used to: use the variational autoencoder to extract the second feature of each video frame in the video data; use the splicing module to splice the second feature of the first video frame in the video data with the second features of other video frames after noise addition to obtain a spliced ​​feature; input the first video frame in the video data into the visual encoder, extract the first feature of the first video frame in the video data, and align them through a linear mapping layer; input the spliced ​​feature and the aligned first feature of the first video frame in the video data into the convolutional encoder-decoder structure for denoising to generate a predicted video; and adjust the parameters of the timing module according to the predicted video and the video data.

[0153] In a possible implementation, the training module 802 is also used to: calculate the loss function value based on other video frames in the video data except the first video frame, and other video frames in the predicted video except the first video frame; and adjust the parameters of the timing module based on the loss function value.

[0154] In a possible implementation, the training module 802 is further used to: perform dimensionality reduction processing on the video data to obtain dimensionality-reduced data; wherein the dimension of the dimensionality-reduced data is the same as the dimension of the image data; and use the dimensionality-reduced data to train the intermediate diffusion model after structural adjustment.

[0155] In a possible implementation manner, an initial value of a parameter of the timing module is zero.

[0156] Based on the same inventive concept of the above method embodiment, the embodiment of the present disclosure further provides a video generation device, which can be used to execute the technical solution described in the above method embodiment. Figure 5 or Figure 7 The steps of the video generation method shown.

[0157] Fig. 9 A structural diagram of a video generating device according to an embodiment of the present disclosure is shown as follows: Fig. 9As shown, the device includes: a second acquisition module 901, used to acquire a reference image; a generation module 902, used to process the reference image using a target model to generate a target video; wherein the target model is a video generation model obtained by any of the above-mentioned model training methods.

[0158] In a possible implementation, the target video includes a preset number of video frames;

[0159] The generation module 902 is also used to: extract the last video frame in the target video; use the last video frame as the reference image; repeatedly process the reference image using the target model to generate the target video, and subsequent operations until the number of video frames in the stitched video reaches a preset threshold, wherein the stitched video is obtained by stitching together the target videos generated by processing each reference image with the target model.

[0160] In a possible implementation, the generation module 902 is also used to: use the visual encoder in the target model to extract the first feature of the reference image, and align it through the linear mapping layer in the target model to obtain the aligned first feature; use the variational autoencoder in the target model to extract the second feature of the reference image, and splice it with the random noise to obtain the spliced ​​feature; input the aligned first feature and the spliced ​​feature into the convolutional encoder-decoder structure in the target model for processing to generate the target video. Figure 8 The model training device shown, Fig. 9 The technical effects and specific descriptions of the video generation device shown and its various possible implementation methods can be found in the above-mentioned model training method and video generation method, which will not be repeated here.

[0161] It should be understood that the division of the modules in the above device is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the modules of the device, wherein the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all modules can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all modules above are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all modules above. All modules of the above device can be realized in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the rest by a hardware circuit.

[0162] In the embodiments of the present disclosure, the processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a neural-network processing unit (NPU), a tensor processing unit (TPU), etc.; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules.

[0163] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above embodiment method, such as: CPU, GPU, NPU, TPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms. In addition, each module in the above device can be fully or partially integrated together, or can be implemented independently, which is not limited.

[0164] An embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of the above embodiment when executing the instructions.

[0165] Fig.10 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.10 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0166] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0167] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0168] An embodiment of the present disclosure provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method in the above embodiment is implemented.

[0169] An embodiment of the present disclosure provides a computer program product, which may include, for example, a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code; when the computer program product runs on a computer, the computer executes the method in the above embodiment.

[0170] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0171] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0172] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0173] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0174] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0175] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0176] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0177] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0178] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that: include: Acquire training data; wherein the training data includes video data and image data; Performing training based on the training data and a pre-trained image diffusion model to obtain a video generation model; Among them, the image diffusion model is a diffusion model that generates images from texts; and the video generation model is a diffusion model that generates videos from images.

2. The method according to claim 1, characterized in that: The training is performed based on the training data and the pre-trained image diffusion model to obtain a video generation model, including: Adjusting the structure of the image diffusion model to obtain an image diffusion model with adjusted structure; Using the image data to train the image diffusion model after the structure adjustment to obtain an intermediate diffusion model, wherein the intermediate diffusion model is a diffusion model for generating images from images; Adjusting the structure of the intermediate diffusion model to obtain an intermediate diffusion model with adjusted structure; The video data is used to train the intermediate diffusion model after the structural adjustment to obtain the video generation model.

3. The method according to claim 2, characterized in that The image diffusion model includes: a text encoder, a convolutional encoder-decoder structure, and a variational autoencoder; The step of adjusting the structure of the image diffusion model to obtain the image diffusion model after the structure adjustment includes: The text encoder is replaced by a visual encoder, and a linear mapping layer is configured after the visual encoder to obtain the image diffusion model after the structure adjustment; The visual encoder is used to extract image features, and the linear mapping layer is used to align the image features with the text features extracted by the text encoder.

4. The method according to claim 3, characterized in that The aligning of the image features with the text features extracted by the text encoder includes: flattening the input features of the global pooling layer in the visual encoder along the spatial dimension to obtain a visual feature sequence, and aligning the length of the visual feature sequence with the length of the text feature.

5. The method according to claim 3, characterized in that: The step of training the image diffusion model after the structure adjustment by using the image data comprises: Inputting the image data into the structure-adjusted image diffusion model, extracting a first feature of the image data through the visual encoder, and extracting a second feature of the image data through the variational autoencoder; Inputting the first feature into a linear mapping layer for alignment processing to obtain an aligned first feature; Inputting the aligned first feature and the second feature into a convolutional encoder-decoder structure for processing to obtain a predicted image; According to the predicted image and the image data, the parameters of the linear mapping layer and each transformer block in the convolutional encoder-decoder structure are adjusted.

6. The method according to claim 3, characterized in that The step of adjusting the structure of the intermediate diffusion model to obtain the intermediate diffusion model after structural adjustment includes: A timing module is added after each transformer block in the convolutional encoder-decoder structure, and a splicing module is added before the convolutional encoder-decoder structure to obtain the intermediate diffusion model after the structure adjustment, wherein the timing module is used to calculate the temporal self-attention to align the content of the first video frame in the video data with the content of each video frame generated by the convolutional encoder-decoder structure; and the splicing module is used to splice each video frame in the video data.

7. The method according to claim 6, characterized in that The step of training the intermediate diffusion model after structural adjustment by using the video data includes: Extracting a second feature of each video frame in the video data using the variational autoencoder; Using the splicing module, the second feature of the first video frame in the video data is spliced ​​with the second features of other video frames after noise is added to obtain a spliced ​​feature; Inputting a first video frame in the video data into the visual encoder, extracting a first feature of the first video frame in the video data, and aligning the first feature through a linear mapping layer; Inputting the spliced ​​features and the first feature of the first video frame in the aligned video data into the convolutional encoder-decoder structure for denoising to generate a predicted video; The parameters of the timing module are adjusted according to the predicted video and the video data.

8. The method according to claim 7, characterized in that The adjusting the parameters of the timing module according to the predicted video and the video data includes: Calculate the loss function value according to other video frames except the first video frame in the video data and other video frames except the first video frame in the predicted video; Based on the loss function value, the parameters of the timing module are adjusted.

9. The method according to claim 3, characterized in that: The step of training the intermediate diffusion model after structural adjustment by using the video data includes: Performing dimensionality reduction processing on the video data to obtain dimensionality-reduced data; wherein the dimensionality of the dimensionality-reduced data is the same as the dimensionality of the image data; The intermediate diffusion model after structural adjustment is trained using the dimension-reduced data.

10. The method according to claim 7, characterized in that The initial values ​​of the parameters of the timing module are zero.

11. A video generation method, characterized in that: The method comprises: Get a reference image; The reference image is processed using a target model to generate a target video; wherein the target model is a video generation model obtained by any one of claims 1-10 above.

12. The method according to claim 11, characterized in that The target video includes a preset number of video frames; The method further comprises: Extract the last video frame in the target video; Using the last video frame as the reference image; Repeat the process of processing the reference image using the target model to generate a target video, and subsequent operations until the number of video frames in the stitched video reaches a preset threshold, wherein the stitched video is obtained by stitching together the target videos generated by processing each reference image with the target model.

13. The method according to claim 11, characterized in that The step of processing the reference image using the target model to generate a target video includes: Extracting a first feature of the reference image using a visual encoder in the target model, and aligning the first feature through a linear mapping layer in the target model to obtain an aligned first feature; Extracting a second feature of the reference image using a variational autoencoder in the target model, and concatenating the second feature with the random noise to obtain a concatenated feature; The aligned first feature and the spliced ​​feature are input into the convolutional encoder-decoder structure in the target model for processing to generate the target video.

14. A model training device, characterized in that: The device comprises: An acquisition module, used to acquire training data; wherein the training data includes video data and image data; A training module is used to perform training based on the training data and a pre-trained image diffusion model to obtain a video generation model; wherein the image diffusion model is a diffusion model that generates images from texts; and the video generation model is a diffusion model that generates videos from images.

15. A video generating device, characterized in that: The device comprises: An acquisition module, used for acquiring a reference image; A generation module is used to process the reference image using a target model to generate a target video; wherein the target model is a video generation model obtained by any one of claims 1-10 above.

16. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 10 when executing the instructions stored in the memory.

17. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images