Verified video generation method, product, equipment and storage medium

Abstract

The invention discloses a text video generation method, a product, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting a text description and a noise vector into a first text video model trained according to a preset distillation strategy, so as to generate a video matched with the text description and a corresponding video implicit vector; the preset distillation strategy is that the historical high-resolution video is de-noised in sequence by using a student model and a diffusion model based on an attention mechanism, and model parameters of the preset student model are updated based on loss corresponding to a de-noising result; performing up-sampling on the video implicit vector, and splicing the obtained sampled vector and the noise vector; and inputting the spliced vector into a second text video model based on the multi-path state space model to generate a target text video. According to the invention, on the basis of the attention mechanism, the multi-path state space model is combined, and through multiple reasoning, the resolution of the text video and the generation efficiency of the text video can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, product, device and storage medium for generating a Vincent video. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC) technology, text-based video models such as SoRA (a large video generation model proposed by OpenAI) have been widely used. This model can generate high-definition videos based on text descriptions, thus providing a new method for rapid prototyping and proof of concept for creative content production, film, animation, game and advertising industries.

[0003] However, current Vincent video models, such as Stable Diffusion Video and Imagen-Video, have a large number of parameters. When using these models for high-resolution video generation, they usually first perform preliminary training on low-resolution videos, and then continuously train super-resolution models to achieve high-resolution image generation. This cascade training method consumes a huge amount of computing power and poses a great challenge to computing resources. In addition, current Vincent video models usually use diffusion models for denoising to complete the generation of video content, which leads to low inference efficiency, especially when generating high-resolution images. The generation speed becomes very slow, so it is necessary to improve the inference speed for high-resolution videos.

[0004] In summary, how to avoid the cumbersome and computationally intensive training process in the training phase to obtain a model for generating high-resolution content, and how to quickly complete the generation of high-resolution video content in the inference phase are issues that need to be urgently addressed. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a method, product, device and storage medium for generating Vincent video, which can improve the resolution of Vincent video and the efficiency of Vincent video generation. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a method for generating a Vincent video, comprising:

[0007] The target text description and the first noise vector are input into the first video model to generate a video matching the target text description and a corresponding first video latent vector; the first video model is a model obtained by training a preset student model using the training set and according to a preset distillation strategy, and the preset distillation strategy is to use the preset student model and the first diffusion model based on the attention mechanism to denoise the historical high-resolution videos in the training set in turn, and update the model parameters of the preset student model based on the loss corresponding to the denoising result;

[0008] Upsampling the first video latent vector, and concatenating the obtained first sampled vector and the second noise vector to obtain a first concatenated vector;

[0009] The first concatenated vector is input into a second text-generated video model to generate a target text-generated video matching the target text description; the second text-generated video model is a model obtained by training a second diffusion model based on a multi-path state space model using a training set.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned Vincent video generation methods when executed by a processor.

[0011] The present application also provides an electronic device, including a processor and a memory; wherein the processor implements the aforementioned Vincent video generation method when executing a computer program stored in the memory.

[0012] The present application also provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned Vincent video generation method is implemented.

[0013] It can be seen that the present application optimizes the traditional distillation strategy, and the optimized distillation strategy undergoes two denoising processes, in which the historical high-resolution videos in the training set are first denoised through the student model, and then the first denoising result is input into the first diffusion model based on the attention mechanism for the second denoising, and the model parameters of the student model are updated based on the losses corresponding to the two denoising results, thereby obtaining a first Vincent video model with high reasoning efficiency; and, on the basis of the diffusion model based on the attention mechanism, the present application combines a second diffusion model based on a multi-path state space model, and by re-inferring the Vincent video output based on the attention mechanism, the resolution of the Vincent video can be further improved; in addition, since the multi-path state space model can only process the features within the specified state space compared to the attention mechanism, the generation efficiency of the Vincent video is improved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0015] Figure 1 A flow chart of a method for generating a Vincent video disclosed in this application;

[0016] Figure 2 A specific distillation training flow chart disclosed in this application;

[0017] Figure 3 A schematic diagram of a specific discriminator structure disclosed in this application;

[0018] Figure 4 A schematic diagram of a specific discriminator submodule structure disclosed in this application;

[0019] Figure 5 A specific second-language video model training flow chart disclosed in this application;

[0020] Figure 6 A schematic diagram of a Wensheng video model structure disclosed in this application, wherein Figure 6 (a) is the traditional diffusion model based on the DIT diffusion module. Figure 6 (b) for Figure 6 (a) Diffusion model after optimization of the DIT diffusion module;

[0021] Figure 7 A schematic diagram of a specific sequence processing unit structure disclosed in this application;

[0022] Figure 8 A specific 4-way state space model scanning schematic diagram disclosed in this application;

[0023] Fig. 9 A schematic diagram of a network structure of a specific state space model disclosed in this application;

[0024] Fig.10 A flow chart of a specific method for generating a Vincent video disclosed in this application;

[0025] Fig.11 This is a flowchart of a specific method for generating Wensheng videos disclosed in this application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0028] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] The present application embodiment discloses a method for generating a viva video. Figure 1 As shown, the method includes:

[0030] Step S11: input the target text description and the first noise vector into the first cultural video model to generate a video matching the target text description and a corresponding first video latent vector; the first cultural video model is a model obtained by training a preset student model using a training set and according to a preset distillation strategy, and the preset distillation strategy is to use a preset student model and a first diffusion model based on an attention mechanism to denoise the historical high-resolution videos in the training set in turn, and update the model parameters of the preset student model based on the loss corresponding to the denoising result.

[0031] In this embodiment, the text description (such as yellow and black tropical fish swimming rapidly in the sea) to be generated by the current user is first obtained to obtain a target text description, and then the target text description and the first noise vector input by the user are input into a first Vincent video model obtained by training a preset student model using a training set and a preset distillation strategy, so as to generate a video matching the target text description and a corresponding first video latent vector through the first Vincent video model. The first noise vector may be a Gaussian noise vector.

[0032] In this embodiment, considering that the student model and the teacher model in the traditional distillation strategy are associated with each other, it is impossible to achieve absolute decoupling. For example, the initial weights and model structures of the student model and the teacher model must be consistent, and the student model must load the initial weights of the teacher model. In this way, the model can only be updated by comparing the forward denoising vectors at similar time steps, resulting in the model obtained after distillation training being less robust. On the other hand, considering that the traditional distillation strategy generates a better student model through equivalent denoising and repeated distillation processes, this will reduce the distillation efficiency and will result in a lower quality of generation of the final student model that meets the inference efficiency. The synthesized samples look blurry and have obvious artifacts. This problem will be more obvious with fewer sampling steps.

[0033] In response to the above problems, this application optimizes the traditional distillation strategy. The optimized distillation strategy is to first use a preset student model and a first diffusion model based on an attention mechanism (such as self-attention mechanism) to denoise the historical high-resolution videos in the training set in turn, and then update the model parameters (such as model weights, etc.) of the preset student model based on the loss corresponding to the denoising result.

[0034] Specifically, the historical high-resolution videos in the training set are denoised in turn using a preset student model and a first diffusion model based on an attention mechanism, and the model parameters of the preset student model are updated based on the loss corresponding to the denoising result, which may include: denoising the historical high-resolution videos in the training set to obtain a first denoised video; inputting the first denoised video into a preset student model for denoising to obtain a first denoised result; the preset student model is a diffusion model with a different network structure from the first diffusion model; inputting the first denoised result and the historical high-resolution video into a preset discriminator for discrimination, and calculating the discrimination loss between the first denoised result and the historical high-resolution video; denoising the first denoised result to obtain a second denoised video; using the first diffusion model as a teacher model, and inputting the second denoised video into the teacher model for denoising to obtain a second denoised result; calculating the distillation loss between the first denoised result and the second denoised result, and calculating the sum of the distillation loss and the discrimination loss to obtain the target loss; guiding the training of the preset student model based on the target loss and using the gradient descent method; the preset student model and the teacher model have different sampling step ranges. In this embodiment, see Figure 2As shown in FIG. 1 , the specific training process of the first cultural video model obtained after training according to the preset distillation strategy is divided into two stages, namely, denoising stage 1 and denoising stage 2. The specific execution process is: first, a training set containing historical text descriptions and corresponding historical high-resolution videos is collected, and then the historical high-resolution videos in the training set are denoised, and the first denoised video obtained after denoising is input into the student model for the first denoising process to obtain the corresponding first denoised result; then, the first denoised result and the historical high-resolution video are input into the discriminator for discrimination, and the discrimination loss between the two is calculated; further, denoising stage 2 is executed, that is, the first denoised result output by the student model is denoised to obtain a second denoised video, and the second denoised video is input into the teacher model (i.e., the first diffusion model based on the attention mechanism) for the second denoising process to obtain the corresponding second denoised result; it should be pointed out that in the entire distillation process, the model parameters of the teacher model (i.e., the first diffusion model) are frozen, that is, the model parameters remain unchanged and the parameters are not updated. Finally, the distillation loss between the two denoised results (i.e., the first denoised result and the second denoised result) is calculated, and the sum of the distillation loss and the discriminant loss is calculated to obtain a target loss. Based on the target loss, the gradient descent method is used to guide the training of the student model, that is, the model parameters are continuously optimized and updated. In addition, the model weights of the discriminator need to be trained during the entire distillation process.

[0035] In addition, the present application decouples the teacher model and the student model. The student model (preset type of diffusion model) and the teacher model (i.e., the first diffusion model) have different network structures, and the sampling step ranges of the student model and the teacher model are also different. For example, the network structure of the teacher model adopts the network structure of the Stable-Diffusion-Video version, while the student model adopts other types of network structures (no specific requirements are made here, and the network weights of the student model can be randomly initialized, or the initial weights of any network can be loaded), and the sampling step range of the student model is {250,500, 750, 1000}, and the sampling step range of the teacher model is {1,…,1000}.

[0036] It can be seen that the optimized distillation strategy performs denoising twice, in which the student model undergoes the first denoising, and the first denoising result is discriminated with the real video to obtain the discriminant loss, and then the first denoised video output by the student model is denoised for the second time by the teacher model to obtain the second denoising result, and then the parameters of the student model are updated based on the discriminant loss and the distillation loss between the two denoising results. Since the result obtained after the teacher model denoising has undergone two denoising processes, the result obtained is very accurate. At this time, the distillation loss is calculated again, and the parameters of the student model can be better updated, so as to obtain a Wensheng video model with high reasoning efficiency, thereby improving the reasoning efficiency of Wensheng video, and thus it is well applied to the scene of generating high-resolution video content. In addition, the optimized distillation strategy adopts an end-to-end denoising distillation method, and only one distillation is required to obtain a qualified student model, which greatly improves the distillation efficiency, so that the model weight after distillation can complete accelerated reasoning, and the reasoning result of the same quality can be obtained in only one step. Moreover, after the above two-stage denoising distillation process, the entire Vincent video reasoning process only requires two denoising steps to obtain a high-resolution Vincent video, thereby greatly improving the reasoning efficiency.

[0037] In a specific embodiment, the first denoised result and the historical high-resolution video are input into a preset discriminator for discrimination, and the discrimination loss between the first denoised result and the historical high-resolution video is calculated. Specifically, it may include: inputting the first denoised result and the historical high-resolution video into a preset discriminator for discrimination; the preset discriminator includes multiple discrimination submodules; calculating the loss between the first denoised result and the historical high-resolution video through each discrimination submodule in the preset discriminator to obtain multiple discrimination losses; calculating the sum of multiple discrimination losses to obtain adversarial loss; calculating the sum of distillation loss and adversarial loss to obtain target loss. In this embodiment, see Figure 3 As shown in Figure 2, the network architecture of the discriminator includes four discriminant submodules, and each discriminant submodule corresponds to a feature extraction network module of a stage. The structure of each discriminant submodule can be found in Figure 4 As shown in the figure, it includes a convolutional layer (Conv layer), a batch normalization layer (BatchNorm layer) and an activation function (such as Leaky ReLU). By presetting each discriminant submodule in the discriminator, the loss between the first denoising result and the historical high-resolution video can be calculated to obtain multiple discriminant losses. Then, all the discriminant losses are superimposed to obtain an adversarial loss, and then the sum of the distillation loss and the adversarial loss is calculated to obtain the target loss for model training.

[0038] In a specific embodiment, the historical high-resolution videos in the training set can be first Perform noise processing to obtain the first noise-added video , the specific calculation formula is:

[0039] ;

[0040] Where s represents the diffusion steps of the student model, which is within the sampling step range of the student model {250, 500, 750, 1000}. and is a hyperparameter for the number of diffusion steps s, is standard Gaussian noise.

[0041] Next, the first noise-added video Input into the student model for denoising to obtain the first denoised video , and the discriminator calculates the first denoised video with historical high-resolution video The discriminant loss between Represents the model parameters of the student model. It should be noted that the first denoised video It is a completely denoised video, not a denoised video in the traditional distillation strategy.

[0042] Furthermore, for the first denoised video Add noise to get the second noisy video , the specific calculation formula is:

[0043] ;

[0044] Where t represents the diffusion steps of the teacher model, which is within the sampling step range of the teacher model {1,…,1000}. and is a hyperparameter for the number of diffusion steps t, is standard Gaussian noise.

[0045] Next, the second noisy video Input the teacher model for the second denoising operation to obtain the second denoised video ; Represents the model parameters of the teacher model. It should be noted that the second denoised video It is a completely denoised video, not a denoised video in the traditional distillation strategy.

[0046] Finally, the denoised video is obtained based on the two previous and subsequent steps. and Calculate the distillation loss, add the distillation loss and the discrimination loss, and use the gradient descent method to complete the training of the student model based on the target loss obtained after the addition.

[0047] In a specific embodiment, when calculating the adversarial loss When , a multi-stage integrated learning method can be adopted and the Hinge Loss function can be used to calculate the loss of the features of the first denoising result and the historical high-resolution video generated by each discriminant submodule in the discriminator at the corresponding stage, so as to obtain multiple discriminant losses and calculate the sum of all discriminant losses. The specific calculation formula is:

[0048] ;

[0049] In the formula, k represents the number of discriminative submodules, represents the discriminant submodule; Represents a video feature extractor. For further information, see Figure 3 As shown in FIG. 1 , after serialization and four stages of feature extraction network modules, the output features of each feature extraction network module are discriminated by the discriminator. By using a multi-stage ensemble learning method to calculate the discriminant loss, the accuracy of the discriminant loss calculation can be further improved.

[0050] Specifically, calculating the distillation loss between the first denoised result and the second denoised result may include: calculating the distillation loss between the first denoised result and the second denoised result based on the first denoised result, the second denoised result and the target hyperparameter and using a mean square error loss function; wherein the target hyperparameter is a hyperparameter for the number of diffusion steps, and the number of diffusion steps is within the range of sampling steps of a preset student model. In this embodiment, the mean square error loss function (MeanSquared Error, mean square error function) can be used to calculate the distillation loss between the first denoised result and the second denoised result, and the specific calculation formula is:

[0051] ;

[0052] In the formula, This shows that the higher the noise level, the smaller the proportion of distillation loss; represents the mean square error loss function.

[0053] Step S12: up-sample the first video latent vector, and concatenate the obtained first sampled vector and the second noise vector to obtain a first concatenated vector.

[0054] In this embodiment, after generating a video matching the target text description and a corresponding first video latent vector, the first video latent vector is further upsampled to obtain a first sampled vector, and then the first sampled vector and the second noise vector are spliced ​​to obtain a first spliced ​​vector. The second noise vector may be a Gaussian noise vector.

[0055] Step S13: input the first concatenated vector into a second text-generated video model to generate a target text-generated video matching the target text description; the second text-generated video model is a model obtained by training a second diffusion model based on a multi-path state space model using a training set.

[0056] In this embodiment, after the first sampled vector and the second noise vector are spliced ​​to obtain a first spliced ​​vector, the first spliced ​​vector can be further input into a second Vincent video model obtained by training a second diffusion model based on a multi-path state space model using a training set, thereby generating a Vincent video matching the target text description and obtaining a target Vincent video with high resolution.

[0057] Specifically, the process of acquiring the second cultural video model may include: collecting historical text descriptions and corresponding historical high-resolution videos, and downsampling the historical high-resolution videos to obtain historical low-resolution videos; compressing the historical high-resolution videos and the historical low-resolution videos from pixel space to latent vector space through a video autoencoder to obtain a compressed high-resolution latent vector and a compressed low-resolution latent vector; splicing the compressed low-resolution latent vector with the third noise vector to obtain a second spliced ​​vector, and inputting the second spliced ​​vector and the historical text description into a second diffusion model based on a multi-path state space model for model training to obtain a second cultural video model. In this embodiment, historical text descriptions and corresponding historical high-resolution videos (i.e., high-resolution video sequences) are first collected to obtain a training set, and then the historical high-resolution videos in the training set are downsampled to obtain corresponding historical low-resolution videos; then, the historical high-resolution videos and historical low-resolution videos are respectively compressed from pixel space to latent vector space by a video autoencoder, such as VideoMAE (video masked autoencoder), to obtain corresponding compressed high-resolution latent vectors and compressed low-resolution latent vectors, and then the compressed low-resolution latent vectors are spliced ​​with a preset noise vector to obtain a spliced ​​vector, and the spliced ​​vector and the historical text descriptions in the training set are input into a diffusion model based on a multi-path state space model for model training, thereby obtaining a second text video model.

[0058] Specifically, concatenating the compressed low-resolution latent vector with the third noise vector to obtain a second concatenated vector may include: upsampling the compressed low-resolution latent vector to generate a vector with the same spatial resolution as the compressed high-resolution latent vector to obtain a second sampled vector; concatenating the second sampled vector with the third noise vector to obtain a second concatenated vector; the third noise vector is a vector obtained by sampling with a Gaussian distribution. Figure 5 As shown, for the high-resolution video sequences in the training set (dimension is ) is downsampled to obtain the corresponding low-resolution video sequence (dimension is ); Then, the high-resolution video sequence and the low-resolution video sequence are compressed from pixel space to latent vector space through the video autoencoder. Here, the video autoencoder has both spatial resolution compression (from 256 to 32) and temporal compression (from 16 to 2), that is, the compression ratio is 8, and the corresponding high-resolution latent vector (dimension is ) and a low-resolution latent vector (of dimension ), the specific compression ratio can be set according to actual needs; then the low-resolution latent vector is upsampled to generate a vector with the same spatial resolution as the high-resolution latent vector, and then the sampled vector is concatenated with the noise vector of the same size obtained after Gaussian distribution sampling to obtain a concatenated vector; the features corresponding to the concatenated vector and the high-resolution latent vector are the concatenated vector features and high-resolution latent vector features Finally, the concatenated vector features and historical text descriptions (such as a spectacular tornado at night) are input into the second diffusion model based on the multi-path state space model for training.

[0059] See also Figure 5 As shown, the training process of the entire second Vincent video model can complete the end-to-end training of high-resolution Vincent videos, thereby avoiding the cascade training method of traditional high-resolution Vincent video models and greatly reducing computing power consumption.

[0060] It should be pointed out that the training process of the second video model specifically includes: obtaining the time component of the second video model; converting the compressed high-resolution latent vector and the second spliced ​​vector into vector features to obtain the target high-resolution latent space vector features and the spliced ​​vector features; based on the time component, the target high-resolution latent space vector features and the spliced ​​vector features, and using the preset loss function to calculate the loss of the second video model, the diffusion loss is obtained. That is, first convert the compressed high-resolution latent vector and the second spliced ​​vector into vector features to obtain the corresponding target high-resolution latent space vector features (i.e. Figure 5 In ) and the concatenated vector features (i.e. Figure 5 In ); Then based on the temporal component of the second video model, the target high-resolution latent space vector features (i.e. ) and the concatenated vector features (i.e. , the dimension is ), and use the preset loss function to calculate the loss of the second video model to get the diffusion loss.

[0061] Specifically, based on the time component, the target high-resolution latent space vector feature and the spliced ​​vector feature, and using a preset loss function to calculate the loss of the second video model, the diffusion loss is obtained, which can include: serializing the spliced ​​vector feature to obtain the serialized feature; encoding the time component and the historical text description respectively to obtain the time step feature and the text feature, and adding the time step feature and the text feature to obtain the conditional feature; inputting the conditional feature and the serialized feature into a sequence processing unit including a multi-channel state space model for state space processing to obtain the first output feature; the sequence processing unit includes a scale layer, a scale plane Shifting layers and multi-layer perceptrons; the multi-layer perceptron is used to perform feature transformation on conditional features, the scale layer is used to perform scale transformation on the features output by the multi-layer perceptron, and the scale shifting layer is used to perform translation transformation on the features output by the multi-layer perceptron; the first output feature is normalized to obtain the second output feature, and the second output feature is deserialized to obtain the predicted noise mean vector; the predicted noise mean vector has the same dimension as the concatenated vector feature; based on the predicted noise mean vector, the concatenated vector feature and the target high-resolution latent space vector feature, the loss of the second video model is calculated using the mean square error loss function to obtain the diffusion loss. In this embodiment, refer to Figure 6 As shown in (b), the concatenated vector features (i.e. Figure 6 The hidden vector in ) to perform serialization operations (including convolution and vector flattening operations) to obtain the serialized feature x (dimension is ); N is the batch size, which can be set to 64; then, the time component (Right now Figure 6 time step vector in ) and historical text description (i.e. Figure 6 The text vector in is encoded to obtain the time step feature and the text feature, and the time step feature and the text feature are added to obtain the conditional feature c (dimension is ),in, ; Further, the conditional feature c and the serialized feature are input into the sequence processing unit containing the multi-channel state space model for state space processing, and then the output features are sequentially subjected to layer normalization operations (such as RMS Norm normalization operations, i.e., root mean square error normalization operations) and deserialization operations, so as to obtain the predicted noise mean vector and the prediction noise standard deviation ; Among them, the predicted noise mean vector and the predicted noise standard deviation are both concatenated with the vector feature Finally, based on the predicted noise mean vector and the concatenated vector features and target high-resolution latent space vector features , and use the mean square error loss function to calculate the loss of the second video model to get the diffusion loss. The specific calculation formula is:

[0062] ;

[0063] In the formula, ; represents the second Vincent video model, represents the model parameters of the second video model; represents the mean square error loss function; Equivalent to predicting the noise mean vector The diffusion loss is calculated according to the above formula to complete the training of the second video model.

[0064] It should be noted that in order to improve the speed of training and reasoning, this application has modified the multi-head attention mechanism in the DIT diffusion module (a diffusion model based on the transformer architecture) in the traditional attention-based diffusion model, turning it into a sequence processing unit that can accelerate the processing of long sequences. Figure 6 As shown, Figure 6 (a) is the traditional diffusion model based on the DIT diffusion module. Figure 6 (b) for Figure 6 The diffusion model after the DIT diffusion module in (a) is optimized, specifically, the DIT diffusion module is replaced by the sequence processing unit.

[0065] For details, see Figure 7 As shown, the sequence processing unit includes a scaling layer, a scaling translation layer and a multilayer perceptron (MLP); the multilayer perceptron is used to perform feature transformation on the conditional feature c, the scaling layer is used to scale the features output by the multilayer perceptron, and the scaled features are input into four one-dimensional convolutions, the outputs of the four one-dimensional convolutions are respectively connected to each state space model in the multi-way state space model (forward state space model, backward state space model, lateral state space model and rear lateral state space model); at the same time, the output of the scaling layer will also pass through the fully connected layer and the activation layer; the scaling translation layer is used to perform a translation transformation on the features output by the multilayer perceptron, and at the same time, it can also perform a translation transformation on the accumulated output features after the convolution of the state space model and the activation layer. Finally, the output of the sequence processing unit is obtained through a fully connected layer. Among them, the output of the sequence processing unit is The dimension of the scale layer is the same as that of the serialized feature x. And the output of the scale-shift layer The calculation formula is:

[0066] ;

[0067] ;

[0068] In the formula, ; and They are the input of the scaling layer and the scaling translation layer respectively.

[0069] It is understandable that the state space model is a recursive model, similar to the RNN (Recurrent Neural Network) network layer. Through the four-way scanning state processing of the input features, the transformer-based two-dimensional attention module can be transformed into a one-dimensional scanning module, thereby reducing the time complexity by one dimension. Figure 8 As shown, Figure 8 It is shown that compared with the attention-based mechanism, the forward state space model, the backward state space model, the lateral state space model, and the rear lateral state space model in the multi-way state space model only need to scan the state space in four directions. The four-way state space model can reduce the time complexity from becomes .

[0070] Of course, any multiple models may be selected from the forward state space model, the backward state space model, the lateral state space model and the rear-lateral state space model according to actual application requirements, or any other multiple state space models may be selected.

[0071] For details, see Fig. 9 As shown in Figure 2, the network structure of each state space model is as follows: Fig. 9 As shown, assuming the input It consists of a series of token vectors and can be expressed as , where L=256; the output of the state space vector model can be expressed as, each token It can be calculated by the following formula:

[0072] ;

[0073] In the formula, ; ; ; ; A represents the state parameter matrix; ; and Both represent fully connected layers. represents the activation layer; represents the current state vector; Represents the previous state vector.

[0074] Specifically, inputting the conditional features and the serialized features into a sequence processing unit including a multi-channel state space model for state space processing to obtain a first output feature may include: position encoding the serialized features by absolute position encoding to obtain encoded features; inputting the conditional features and the encoded features into a sequence processing unit including a multi-channel state space model for state space processing to obtain a first output feature; the first output feature has the same dimension as the serialized feature. In this embodiment, see Figure 6 As shown, considering Figure 6 The other modules in do not contain convolution operations, so a position encoding vector is added after the serialization operation. Specifically, the absolute position encoding method can be used (the specific position encoding method is not limited and can be selected according to actual conditions) to position encode the serialized features, and then the conditional features c and the encoded features are input into the sequence processing unit containing a multi-way state space model for state space processing; wherein, the output features of the sequence processing unit are consistent with the dimensions of the serialized features.

[0075] Specifically, encoding the time component and the historical text description respectively to obtain the time step feature and the text feature may include: inputting the time component into the time embedder for feature encoding to obtain the time step feature; inputting the historical text description into the text embedder for encoding to obtain the text feature. For example, using the time embedder in the same way as DIT to convert the time component into the time step feature (dimension is ), and uses the text embedding of the T5-XXL text encoder type, and obtains a dimension of The text features of .

[0076] It can be seen that the embodiment of the present application optimizes the traditional distillation strategy, and the optimized distillation strategy undergoes two denoising processes, wherein the historical high-resolution videos in the training set are first denoised through the student model, and then the first denoising result is input into the first diffusion model based on the attention mechanism for a second denoising, and the model parameters of the student model are updated based on the losses corresponding to the two denoising results, thereby obtaining a first Vincent video model with high reasoning efficiency; and, on the basis of the diffusion model based on the attention mechanism, the embodiment of the present application combines the second diffusion model based on the multi-path state space model, and the resolution of the Vincent video output based on the attention mechanism is further improved by re-inferring the Vincent video; in addition, since the multi-path state space model can only process the features in the specified state space compared to the attention mechanism, the generation efficiency of the Vincent video is improved, thereby improving the user experience.

[0077] The present application embodiment discloses a specific method for generating a Vincent video. Fig.10 As shown, the method includes:

[0078] Step S21: input the target text description and the first noise vector into the first cultural video model to generate a video matching the target text description and a corresponding first video latent vector; the first cultural video model is a model obtained by training a preset student model using a training set and according to a preset distillation strategy, and the preset distillation strategy is to use a preset student model and a first diffusion model based on an attention mechanism to denoise the historical high-resolution videos in the training set in turn, and update the model parameters of the preset student model based on the loss corresponding to the denoising result.

[0079] Step S22: up-sample the first video latent vector, and concatenate the obtained first sampled vector and the second noise vector to obtain a first concatenated vector.

[0080] Step S23: input the first concatenated vector into a second video model to generate a video matching the target text description and a corresponding second video latent vector; the second video model is a model obtained by training a second diffusion model based on a multi-path state space model using a training set.

[0081] In this embodiment, see Fig.11 As shown, after obtaining the first concatenated vector, it can be input into the second text-generated video model obtained by training the second diffusion model based on the multi-path state space model using the training set, thereby generating a video matching the target text description and the corresponding second video latent vector.

[0082] Step S24: Decode the second video latent vector through a video decoder to obtain a target text video corresponding to the target text description.

[0083] In this embodiment, see Fig.11 As shown, the second video latent vector can be decoded by a video decoder to obtain a high-resolution target text video corresponding to the target text description.

[0084] For more specific processing procedures of the above steps S21 and S22, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0085] It can be seen that the embodiment of the present application can obtain a high-resolution Vincent video by decoding the second video latent vector through a video decoder. In addition, the present application combines a diffusion model based on a multi-path state space model on the basis of a diffusion model based on an attention mechanism, and the diffusion model based on an attention mechanism adopts an optimized distillation strategy for training. The strategy only requires two denoising steps to obtain a high-resolution Vincent video, thereby greatly improving the reasoning efficiency and thus improving the user experience.

[0086] The embodiments of the present application also provide a Wensheng video generation device. The description of the features in the embodiments corresponding to the device can be found in the relevant description of the embodiments corresponding to the process recovery method, which will not be repeated here.

[0087] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the Vincent video generation method.

[0088] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned embodiments of the Vincent video generation method when running.

[0089] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0090] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the Vincent video generation method are implemented.

[0091] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the Vincent video generation method are implemented.

[0092] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0093] The above is a detailed introduction to a method, product, device and storage medium for generating a video according to the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A method for generating a Vincent video, characterized in that: include: Input the target text description and the first noise vector into the first cultural video model to generate a video matching the target text description and a corresponding first video latent vector; the first cultural video model is a model obtained by training a preset student model using a training set and according to a preset distillation strategy, wherein the preset distillation strategy is to use the preset student model and a first diffusion model based on an attention mechanism to sequentially denoise the historical high-resolution videos in the training set, and update the model parameters of the preset student model based on the loss corresponding to the denoising result; Upsampling the first video latent vector, and concatenating the obtained first sampled vector and the second noise vector to obtain a first concatenated vector; Inputting the first concatenated vector into a second language video model to generate a target language video matching the target text description; The second video model is a model obtained by training a second diffusion model based on a multi-path state space model using the training set.

2. The method for generating a Vincent video according to claim 1, characterized in that: Before inputting the target text description and the first noise vector into the first text-generated video model, the method further includes: Collecting historical text descriptions and corresponding historical high-resolution videos, and downsampling the historical high-resolution videos to obtain historical low-resolution videos; Compressing the historical high-resolution video and the historical low-resolution video from pixel space to latent vector space respectively by using a video autoencoder to obtain a compressed high-resolution latent vector and a compressed low-resolution latent vector; The compressed low-resolution latent vector is spliced ​​with the third noise vector to obtain a second spliced ​​vector, and the second spliced ​​vector and the historical text description are input into a second diffusion model based on a multi-channel state space model for model training to obtain the second cultural video model.

3. The method for generating a Vincent video according to claim 2, characterized in that: The step of concatenating the compressed low-resolution latent vector with the third noise vector to obtain a second concatenated vector comprises: Upsampling the compressed low-resolution latent vector to generate a vector having the same spatial resolution as the compressed high-resolution latent vector, to obtain a second sampled vector; The second sampled vector is concatenated with the third noise vector to obtain a second concatenated vector; the third noise vector is a vector obtained after Gaussian distribution sampling.

4. The method for generating a Vincent video according to claim 2, characterized in that: The training process of the second video model also includes: Obtaining a time component of the second video model; Converting the compressed high-resolution latent vector and the second concatenated vector into vector features to obtain target high-resolution latent space vector features and concatenated vector features; Based on the time component, the target high-resolution latent space vector feature and the spliced ​​vector feature, a loss of the second video model is calculated using a preset loss function to obtain a diffusion loss.

5. The method for generating a Vincent video according to claim 4, characterized in that: The method of calculating the loss of the second video model based on the time component, the target high-resolution latent space vector feature and the spliced ​​vector feature by using a preset loss function to obtain the diffusion loss includes: Performing a serialization operation on the concatenated vector features to obtain serialized features; Encoding the time component and the historical text description respectively to obtain a time step feature and a text feature, and adding the time step feature and the text feature to obtain a conditional feature; The conditional features and the serialized features are input into a sequence processing unit including a multi-channel state space model for state space processing to obtain a first output feature; the sequence processing unit includes a scaling layer, a scaling translation layer and a multi-layer perceptron; the multi-layer perceptron is used to perform feature transformation on the conditional features, the scaling layer is used to perform scaling transformation on the features output by the multi-layer perceptron, and the scaling translation layer is used to perform translation transformation on the features output by the multi-layer perceptron; Normalizing the first output feature to obtain a second output feature, and deserializing the second output feature to obtain a predicted noise mean vector; the predicted noise mean vector has the same dimension as the concatenated vector feature; The loss of the second video model is calculated based on the predicted noise mean vector, the concatenated vector features and the target high-resolution latent space vector features using a mean square error loss function to obtain a diffusion loss.

6. The method for generating a Vincent video according to claim 5, characterized in that: The encoding of the time component and the historical text description respectively to obtain a time step feature and a text feature includes: Inputting the time component into a time embedder for feature encoding to obtain a time step feature; The historical text description is input into a text embedder for encoding to obtain text features.

7. The method for generating a Vincent video according to claim 5, characterized in that: The step of inputting the conditional features and the serialized features into a sequence processing unit including a multi-path state space model for state space processing to obtain a first output feature comprises: Position encoding is performed on the serialized features by means of absolute position encoding to obtain encoded features; The conditional features and the encoded features are input into a sequence processing unit including a multi-way state space model for state space processing to obtain a first output feature; the first output feature has the same dimension as the serialized feature.

8. The method for generating a Vincent video according to claim 2, characterized in that: The multi-path state space model includes any multiple models among a forward state space model, a backward state space model, a lateral state space model and a rear-lateral state space model.

9. The method for generating a Vincent video according to claim 1, characterized in that: The method of sequentially denoising the historical high-resolution videos in the training set by using the preset student model and the first diffusion model based on the attention mechanism, and updating the model parameters of the preset student model based on the loss corresponding to the denoising result, includes: Performing noise processing on the historical high-resolution video in the training set to obtain a first noisy video; Inputting the first noisy video into a preset student model for denoising to obtain a first denoising result; the preset student model is a diffusion model with a different network structure from the first diffusion model; Inputting the first denoised result and the historical high-resolution video into a preset discriminator for discrimination, and calculating the discrimination loss between the first denoised result and the historical high-resolution video; Performing noise processing on the first denoising result to obtain a second noisy video; Using the first diffusion model as a teacher model, and inputting the second noisy video into the teacher model for denoising, to obtain a second denoised result; Calculating a distillation loss between the first denoised result and the second denoised result, and calculating a sum of the distillation loss and the discrimination loss to obtain a target loss; The training of the preset student model is guided based on the target loss and using the gradient descent method; the preset student model and the teacher model have different sampling step ranges.

10. The method for generating a Vincent video according to claim 9, characterized in that: The step of inputting the first denoised result and the historical high-resolution video into a preset discriminator for discrimination, and calculating the discrimination loss between the first denoised result and the historical high-resolution video includes: Inputting the first denoising result and the historical high-resolution video into a preset discriminator for discrimination; the preset discriminator includes a plurality of discrimination submodules; Calculating the loss between the first denoising result and the historical high-resolution video through each of the discriminant submodules in the preset discriminator to obtain a plurality of discriminant losses; Calculate the sum of the multiple discriminant losses to obtain the adversarial loss; Accordingly, the calculation of the sum of the distillation loss and the discrimination loss to obtain the target loss includes: The sum of the distillation loss and the adversarial loss is calculated to obtain the target loss.

11. The method for generating a Vincent video according to claim 9, characterized in that: The calculating the distillation loss between the first denoised result and the second denoised result includes: Calculate the distillation loss between the first denoised result and the second denoised result based on the first denoised result, the second denoised result and the target hyperparameter and using a mean square error loss function; The target hyperparameter is a hyperparameter for the number of diffusion steps, and the number of diffusion steps is within the range of sampling steps of the preset student model.

12. The method for generating a Vincent video according to any one of claims 1 to 11, characterized in that: The step of inputting the first concatenated vector into a second language video model to generate a target language video matching the target text description includes: Inputting the first concatenated vector into a second text-generated video model to generate a video matching the target text description and a corresponding second video latent vector; The second video latent vector is decoded by a video decoder to obtain a target text video corresponding to the target text description.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the Vincent video generation method as claimed in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the Vincent video generation method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the Vincent video generation method according to any one of claims 1 to 12 are implemented.

Images