Training method of variational auto-encoder, video generation method and corresponding device

Abstract

The embodiment of the invention discloses a training method of a variational auto-encoder, a video generation method and a corresponding device. According to the main technical scheme, the method comprises the steps of obtaining training data comprising a plurality of video samples; training a variational auto-encoder by using the training data; wherein the training comprises the following steps: respectively inputting a video sample into a variational auto-encoder and a video basic model, and obtaining a hidden space tensor output by an encoder in the variational auto-encoder and a first tensor output by the video basic model; mapping the hidden space tensor into a second tensor, wherein the shape of the second tensor is the same as that of the first tensor; and determining a value of a loss function by using the first tensor and the second tensor, and updating model parameters of the variational auto-encoder by using the value of the loss function. By means of the mode, the variational auto-encoder can inherit semantic priori of the video basic model, the semantic understanding capacity of the variational auto-encoder is improved, and then the video generation quality based on the variational auto-encoder is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and computer vision technology, and in particular to a training method for a variational autoencoder, a video generation method, and corresponding devices. Background Art

[0002] In recent years, with the rapid development of deep learning, AI-generated content (AIGC) has achieved breakthroughs. Diffusion-based image and video technologies have become mainstream research and application areas due to their excellent generation quality and diversity. Diffusion models transform a noisy tensor into media content (such as an image or video) that matches the user's description through a gradual denoising process.

[0003] To reduce the training and inference costs of pixel-space denoising, a technique for latent space denoising within a VAE (Variational AutoEncoder) was proposed. This technique performs denoising based on the latent space tensor extracted by the VAE. However, the latent space tensor currently extracted by VAE primarily contains low-level information about the media content and lacks semantics, resulting in poor quality in the resulting video. Summary of the Invention

[0004] In view of this, the present application provides a VAE training method, a video generation method and a corresponding device to improve the semantics of the latent space tensor extracted by VAE, thereby improving the quality of video generation.

[0005] This application provides the following solutions: In a first aspect, a method for training a VAE is provided, wherein the VAE includes an encoder and a decoder; the method includes: Obtaining training data including a plurality of video samples; The VAE is trained using the training data; wherein the training includes: inputting the video samples into the VAE and the video base model respectively, obtaining the latent space tensor output by the encoder in the VAE and the first tensor output by the video base model; mapping the latent space tensor into a second tensor, the second tensor having the same shape as the first tensor; determining the value of a first loss function term using the difference index value between the first tensor and the second tensor; determining the value of the loss function using at least the value of the first loss function term, and updating the model parameters of the VAE using the value of the loss function.

[0006] According to an implementable manner in an embodiment of the present application, obtaining the latent space tensor output by the encoder in the VAE and the first tensor output by the video base model includes: obtaining the latent space tensors respectively output by multiple encoding network layers in the encoder, and obtaining the first tensors respectively output by multiple encoding network layers in the video base model; Mapping the latent space tensor into a second tensor includes: mapping the latent space tensors respectively output by a plurality of encoding network layers in the encoder to obtain a plurality of second tensors; The value of the first loss function item is determined by using the difference index value between the first tensor and the second tensor, including: determining the difference index values between the multiple second tensors and the first tensor output by the corresponding encoding network layer in the video basic model, and performing weighted summation on the determined multiple difference index values to obtain the value of the first loss function item.

[0007] According to an implementable manner in an embodiment of the present application, determining the value of the loss function by at least using the value of the first loss function term includes: The value of the loss function is determined using the value of the first loss function item and the value of the second loss function item, wherein the value of the second loss function item is obtained by using the difference between the first restored video and the video sample, and the first restored video is obtained by the decoder decoding using the latent space tensor.

[0008] According to an achievable method in an embodiment of the present application, the training further includes: performing the same affine transformation on the video sample and the latent space tensor, inputting the latent space tensor obtained after the affine transformation into the decoder, and obtaining a second restored video decoded by the decoder; The method of determining the value of the loss function by at least using the value of the first loss function item includes: determining the value of the loss function item by using the value of the first loss function item and the value of the third loss function item, wherein the value of the third loss function item is obtained by using the difference between the second restored video and the video sample obtained after the affine transformation.

[0009] According to an achievable method in an embodiment of the present application, the training further includes: obtaining power spectra of the latent space tensor and the latent space tensor after the affine transformation in the frequency domain; The determining the value of the loss function term by using the value of the first loss function term and the value of the third loss function term includes: The value of the loss function term is determined using the value of the first loss function term, the value of the third loss function term, and the value of the fourth loss function term, wherein the fourth loss function term is obtained by using the difference in high-frequency components between the power spectrum corresponding to the latent space tensor and the power spectrum corresponding to the latent space tensor after the affine transformation.

[0010] In a second aspect, a video generation method is provided, the method comprising: Obtain control conditions; Use the encoder in VAE to obtain the latent space tensor of the noisy video; Denoising the latent space tensor of the noisy video using the control condition to obtain a denoised latent space tensor; Decoding the denoised latent space tensor using the VAE decoder to obtain a target video; Wherein, the VAE is pre-trained using the method described in the first aspect above.

[0011] In a third aspect, a VAE training device is provided, wherein the VAE includes an encoder and a decoder; the device includes: A sample acquisition unit, configured to acquire training data including a plurality of video samples; A model training unit is configured to train the VAE using the training data; wherein the training includes: inputting the video samples into the VAE and the video base model respectively, obtaining the latent space tensor output by the encoder in the VAE and the first tensor output by the video base model; mapping the latent space tensor into a second tensor, the second tensor having the same shape as the first tensor; determining the value of a first loss function term using a difference index value between the first tensor and the second tensor; determining the value of a loss function using at least the value of the first loss function term, and updating the model parameters of the VAE using the value of the loss function.

[0012] In a fourth aspect, a video generation device is provided, the device comprising: a condition acquisition unit configured to acquire a control condition; The video generation unit is configured to obtain a latent space tensor of a noisy video using an encoder in a VAE; denoise the latent space tensor of the noisy video using the control condition to obtain a denoised latent space tensor; and decode the denoised latent space tensor using a decoder of the VAE to obtain a target video; wherein the VAE is pre-trained using the method described in the first aspect above.

[0013] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the first and second aspects are implemented.

[0014] According to a sixth aspect, an electronic device is provided, including: one or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first and second aspects above.

[0015] In a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first and second aspects above.

[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects: 1) In this application, a video base model with stronger semantic understanding ability is used as a "teacher model" to train VAE so that the latent space tensor output by VAE is "aligned" with the first tensor output by the video base model. Guided by the video base model as a feature, VAE can inherit the semantic prior of the video base model, improve the semantic understanding ability of VAE, and thus improve the quality of video generation based on VAE.

[0017] 2) By aligning the features of multiple encoding network layers in the VAE with those in the video base model layer by layer, this application enables the VAE encoder to more completely learn the semantics from the video base model, from basic to abstract, avoiding the loss of valuable low-level details. Furthermore, this layer-by-layer feature alignment provides the VAE encoder with a denser supervisory signal. The gradient feedback at each layer helps the model update parameters more stably, thereby accelerating training convergence.

[0018] 3) In this application, a second loss function term can be derived using the reconstruction loss and the difference between the first restored video and the video sample. This loss function is then combined with the first loss function term to determine its value. This second loss function term essentially uses self-supervised learning with unlabeled data to constrain the "authenticity" of the VAE output. Combined with the first loss function term, this multi-task learning method improves the semantics of the VAE while also ensuring the model's generalization performance.

[0019] 4) This application adds perturbations to the latent space tensor output by the VAE encoder, i.e., performs an affine transformation, and then uses the decoder to reconstruct the video to obtain a second restored video, so that it is as consistent as possible with the result of adding the same perturbations to the video samples. This guided training can enhance the uniformity of the VAE's distribution in the latent space, allowing the VAE to capture more subtle differences in the training data, generate video details that are closer to the real scene, and make the features of adjacent frames closer in space, ensuring that the movement of objects and changes in lighting in the video conform to physical laws, reducing flickering or unnatural mutations.

[0020] 5) In this application, the difference in high-frequency components between the power spectrum corresponding to the latent space tensor output by the VAE encoder and the power spectrum corresponding to the latent space tensor after affine transformation is used to determine the fourth loss function term. The fourth loss function term penalizes the high-frequency energy difference, allowing the loss to focus on the difference in high-frequency components, and ultimately achieving the effect of detail preservation and artifact suppression in video reconstruction.

[0021] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 Flowchart of the VAE training method provided in the embodiment of this application.

[0024] Figure 2 A schematic diagram of the principles of training a VAE provided in an embodiment of the present application.

[0025] Figure 3 A schematic diagram of another principle for training VAE provided in an embodiment of the present application.

[0026] Figure 4 A schematic diagram of the principle of a Vincent video provided in an embodiment of the present application.

[0027] Figure 5 2 is a system architecture diagram applicable to the embodiments of the present application.

[0028] Figure 6 A schematic block diagram of a VAE training device provided in an embodiment of the present application.

[0029] Figure 7A schematic block diagram of a video generation device provided in an embodiment of the present application.

[0030] Figure 8 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0032] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0033] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0034] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0035] Several technologies have been proposed for denoising the latent space of VAEs, significantly reducing training and inference costs. Common VAE supervision loss functions include reconstruction loss and adversarial loss. Under the supervision of these loss functions, the latent space of VAEs primarily contains low-level information in the video, lacking semantics, resulting in poor quality of the generated video.

[0036] In view of this, this application provides a new VAE training idea. Figure 1 This is a flow chart of the VAE training method provided in the embodiment of this application. Figure 1 As shown in , the method may include the following steps: Step 101: Obtain training data including multiple video samples.

[0037] Step 103: Train the VAE using the above training data; wherein the training includes: inputting the video samples into the VAE and the video base model respectively, obtaining the latent space tensor output by the encoder in the VAE and the first tensor output by the video base model; mapping the latent space tensor into a second tensor, the second tensor having the same shape as the first tensor; determining the value of the first loss function term using the difference index value between the first tensor and the second tensor; determining the value of the loss function using at least the value of the first loss function term, and updating the model parameters of the VAE using the value of the loss function.

[0038] It can be seen from the above process that in this application, a video base model with stronger semantic understanding ability is used as a "teacher model" to train VAE so that the latent space tensor output by VAE is "aligned" with the first tensor output by the video base model. The video base model is used as a feature guide, so that VAE can inherit the semantic prior of the video base model, improve the semantic understanding ability of VAE, and thus improve the quality of video generation based on VAE.

[0039] The following describes in detail step 103 in the above process and the effects that can be further produced, with reference to an embodiment. It should be noted that the terms "first" and "second" in this disclosure do not restrict size, order, or quantity, but are merely used to distinguish between them. For example, "first tensor" and "second tensor" are used to distinguish between two tensors.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] like Figure 2 As shown in , in an embodiment of the present application, the same video sample is input into the VAE and the video base model respectively. The VAE includes an encoder and a decoder.

[0042] Video base models are a type of foundational model specifically designed for processing and understanding video data. They are pre-trained using large-scale video data and are designed to learn universal representations for various video understanding tasks. In the embodiments of this application, the video base model can be, but is not limited to, VideoMAE-2, InternVideo, VidCLIP, and others.

[0043] You can select a pre-trained video base model as a "teacher model." You can use this model to perform video recognition or classification tasks, and use metrics to measure the effectiveness of these tasks. For example, the video base model that meets the pre-set metrics, such as classification accuracy and recognition accuracy, is selected as the "teacher model."

[0044] The video sample passes through the VAE encoder to output the latent space tensor .in, Indicates the batch size, that is, the number of video samples processed in parallel. is the number of channels, usually feature dimensions or color channels, etc. is the time step, usually the number of video frames. Height, which represents the vertical size of the video frame. is the width, which indicates the horizontal size of the video frame.

[0045] In the embodiment of the present application, a lightweight projection head can be constructed , for example, using an upsampling network. The projection head can output the latent space tensor of the VAE encoder The projection is a tensor with the same shape as the second tensor of the "teacher model", that is, the video base model. The tensor obtained after projection is called the second tensor, that is, the second tensor It can be: In an embodiment of the present application, the difference index value between the first tensor and the second tensor can be used to determine the value of the first loss function term; at least the value of the loss function term can be used to determine the value of the loss function.

[0046] Among them, the first loss function term is used to characterize the difference between the first tensor and the second tensor. The difference index value can be designed using the cosine similarity between the first tensor and the second tensor, the bidirectional KL divergence between dimensions, etc. For example, the first loss term can use the alignment loss express: in, It reflects the second tensor and the first tensor The similarity between them is calculated. Since the input is a video sample, the corresponding latent space tensor is essentially a spatiotemporal feature. Therefore, the spatiotemporal average cosine similarity between the first tensor and the second tensor can be used. When calculating the spatiotemporal average similarity between the first tensor and the second tensor, the two tensors can be first expanded according to the spatiotemporal dimension, and the channel vector of each spatiotemporal position is calculated. Then, the cosine similarity between the channel vectors of the first tensor and the second tensor at each spatiotemporal position is calculated separately. The cosine similarities at all spatiotemporal positions are then averaged to obtain the spatiotemporal average cosine similarity.

[0047] Since the encoder of VAE usually includes multiple encoding network layers, the video base model also includes multiple encoding network layers, and the features output by different network layers often correspond to different levels of semantics, such as the low-level network encoding basic features such as edges and textures, and the high-level network encoding semantics, overall structure and other features. In the embodiment of the present application, the encoder of VAE and the video base model can also be aligned layer by layer. Specifically, the latent space tensors respectively output by multiple encoding network layers in the encoder of VAE can be obtained, and the first tensors respectively output by multiple encoding network layers in the video base model can be obtained; the latent space tensors respectively output by multiple encoding network layers in the encoder of VAE are mapped to obtain multiple second tensors. When determining the value of the first loss term, the difference index values between the multiple second tensors and the first tensor output by the corresponding encoding network layer in the video base model can be determined, and the determined multiple difference index values are weighted summed to obtain the value of the first loss function term.

[0048] If the number of coding network layers in the VAE encoder is consistent with the number of coding network layers in the video base model, the difference index value can be calculated layer by layer. If the number of coding network layers in the VAE encoder is inconsistent with the number of coding network layers in the video base model, for example, the VAE encoder has 10 coding network layers, while the video base model has 30 coding network layers, then some layers can be selected from the video base model to correspond one by one to the coding network layers in the VAE encoder, for example, every 3 layers in the video base model corresponds to one coding network layer in the VAE encoder. Assume that a total of layer, the first loss term can use the following alignment loss express: in, Indicates that the first The second tensor obtained by projecting the latent space tensor output by the layer encoding network layer, Indicates the first The first tensor output by the encoding network layer in the video base model corresponding to the layer encoding network layer, Indicates the The weight coefficient corresponding to the layer.

[0049] This layer-by-layer feature alignment allows the VAE encoder to more completely learn the semantics from the underlying video model, from basic to abstract, avoiding the loss of valuable low-level details. Furthermore, layer-by-layer feature alignment provides the VAE encoder with denser supervisory signals. Gradient feedback at each layer helps the model update parameters more stably, accelerating training convergence.

[0050] When determining the loss function, in addition to the first loss function term, a second loss function term may be further combined. The second loss function term may adopt the reconstruction loss. Figure 2 As shown in , after a video sample is input into a VAE, the VAE encoder generates a latent space tensor. The decoder then decodes the latent space tensor to reconstruct the video, thereby obtaining a restored video, referred to herein as the first restored video. The second loss function term can then reflect the difference between the first restored video and the video sample. The second loss function term can use L1 loss and / or SSIM loss.

[0051] L1 loss: Also known as mean absolute error (MAE), it calculates the average of the absolute differences between the corresponding pixel values in the first restored video and the video sample. The advantage of L1 loss is that it is relatively insensitive to outliers and can ensure that the reconstructed result (the first restored video) is generally close to the true value (the video sample), helping to preserve the basic structure and contour information of the video. For example, in video denoising tasks, L1 loss can ensure that the denoised video maintains the general shape and structure of the original video.

[0052] SSIM loss: The SSIM loss evaluates video quality by comparing video similarities in brightness, contrast, and structural information. It is based on the perception of the human visual system and typically evaluates local regions of the image within a video. In video reconstruction, SSIM measures the structural similarity between the reconstructed video and the original, prioritizing the preservation of structural features such as texture and edges. For example, for a video containing complex textures, SSIM can better reflect whether the reconstructed video accurately restores these texture details.

[0053] L1 loss combined with SSIM: Combining the L1 and SSIM losses for video reconstruction leverages the strengths of both. The L1 loss ensures that the overall pixel value difference between the reconstructed video and the original video is minimal, while SSIM focuses on optimizing the structural similarity of the video, making the reconstruction visually closer to the original and preserving more detail and texture information. This combination produces clearer and more realistic reconstructed videos.

[0054] The above reconstruction loss is essentially the use of self-supervised learning of unlabeled data to constrain the "authenticity" of the VAE output. Combined with the first loss function term, it forms multi-task learning, which improves the semantics of VAE while also ensuring the generalization performance of the model.

[0055] In addition, the embodiments of the present application also provide further improvements. Figure 3 As shown in , for the latent space tensor output by the encoder, before inputting it into the decoder, the latent space tensor can be first Perform affine transformation to obtain the latent space tensor after affine transformation . It can be expressed as: Among them, Affine() represents affine transformation. The so-called affine transformation is a mapping that can keep the affine combination unchanged. In space, affine transformation can be expressed as a composite operation of "linear transformation + translation", and linear transformation includes scaling and / or rotation. In the embodiment of the present application, the latent space tensor output by the encoder is The affine transformation can be random or regular. For example, randomly sampling the scaling ratio, translation amplitude and rotation amplitude within the preset scaling range, translation range and rotation range, and then performing the transformation on the latent space tensor. Perform an affine transformation. The scaling range, translation range, and rotation range can be pre-set. For example, the scaling range can be [0.7, 1.3], which means that the scaling ratio can be randomly selected from 0.7 to 1.3.

[0056] The decoder is based on the latent space tensor after affine transformation Decode and reconstruct the video. The decoded video is referred to as the second restored video. It can be expressed as: in, Represents the processing function of the decoder.

[0057] In this embodiment, the video samples The same affine transformation is also performed to obtain the video sample after affine transformation , which can be expressed as: The third loss function also uses reconstruction loss, but the reconstruction loss is based on the second restored video And the video samples after affine transformation The third loss function term can be obtained by loss and / or SSIM loss.

[0058] In this way, the value of the loss function used to train the VAE can be obtained by using the value of the first loss function term and the value of the second loss function term, for example, after weighted summation.

[0059] The above-mentioned affine transformation process is essentially a process of adding perturbations. That is, by adding perturbations to the latent space tensor output by the VAE encoder and then reconstructing the video using the decoder, the second restored video needs to be as consistent as possible with the result of adding the same perturbations to the video sample. This guided training can enhance the uniformity of the VAE's distribution in the latent space, allowing the VAE to capture more subtle differences in the training data, generate video details that are closer to the real scene, and make the features of adjacent frames closer in space, ensuring that the motion of objects and changes in lighting in the video conform to physical laws and reducing flickering or unnatural mutations.

[0060] For signals such as video, the low-frequency components in the frequency domain correspond to the overall structure of the signal, such as the large outline of the video frame, and the high-frequency components correspond to details, edges, and textures. Therefore, in order to further improve the effect of video reconstruction, the latent space tensor output by the VAE encoder can be And the latent space tensor after the above affine transformation Converted to the frequency domain, we get and The power spectrum in the frequency domain. Then use and The difference in high-frequency components between the power spectra in the frequency domain determines the fourth loss function term, such as the fourth loss function term The following formula can be used to punish and The high-frequency energy difference: in, is a processing function for extracting high-frequency components. and After Fourier transform, high-frequency components are extracted through a high-pass filter.

[0061] The fourth loss function term mentioned above can penalize high-frequency energy differences, allowing the loss to focus on the differences in high-frequency components, ultimately achieving the effects of detail preservation and artifact suppression in video reconstruction.

[0062] During the actual training process, multiple loss function terms can be weighted and summed to obtain the value of the loss function. For example, the first loss function term and the second loss function term can be weighted and summed to obtain the value of the loss function. For another example, the first loss function and the third loss function term can be weighted and summed to obtain the value of the loss function. For another example, the first loss function, the third loss function term and the fourth loss function term can be weighted and summed to obtain the value of the loss function. The value of the loss function is then used to update the model parameters of the VAE (i.e., the encoder and decoder included in the VAE) using methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, etc.

[0063] It should be noted that in the above training process, the video base model serves as an auxiliary "teacher model". Its model parameters are frozen during the entire VAE training process and will not be updated. Only the VAE model parameters are updated. In addition, the projection head and affine transformation processing involved in the above training process are only used during the VAE training process. After the VAE training is completed, the projection head and affine transformation processing are not used in the process of video generation using VAE combined with the diffusion model. In other words, the above-mentioned projection head and affine transformation processing are used as auxiliary training during the VAE training process, and are not used in the actual inference process.

[0064] After the VAE training is completed, the video can be generated based on the trained VAE and combined with the diffusion model. Figure 4 As shown in , first, a control condition is obtained, where the control condition can be unimodal or multimodal. For example, if it is unimodal, it can be a text description. For another example, if it is multimodal, it can be a text description and a reference image.

[0065] The encoder in the VAE is used to obtain the latent space tensor of the noisy video, which is then fed into the diffusion model. The diffusion model denoises the latent space tensor based on a control condition, which includes noise predictions for T time steps. The noise predicted over these T time steps is then used to denoise the latent space tensor, where T is a preset positive integer.

[0066] The denoised latent space tensor is input into the VAE decoder, and the VAE decoder performs decoding processing based on the denoised latent space tensor to obtain the target video.

[0067] Take Vincent's video as an example. Figure 4As shown in . The user inputs a description text as a control condition, and the description text is passed through a text encoder to obtain a feature representation of the description text. The noisy video is passed through a VAE encoder to obtain a latent space tensor. The diffusion model uses the feature representation of the description text as a guide to denoise the latent space tensor. The denoised latent space tensor is denoised by a VAE decoder to obtain the target video. The diffusion model can be implemented using networks such as U-Net and U-Vit, and this application does not limit the implementation method of the diffusion model.

[0068] If the control condition includes other modalities, such as descriptive text and a reference image, in addition to using the text encoder to encode the descriptive text to obtain a feature representation of the descriptive text, the image encoder can also be used to encode the reference image to obtain a feature representation of the reference image. The feature representations of the descriptive text and the reference image are then fused and input into the diffusion model as guidance.

[0069] Figure 5 This is the system architecture diagram applicable to this application, such as Figure 5 As shown in , the system architecture may include: user equipment, a video generation device located on the server side, and a VAE training device.

[0070] The user equipment and the server can communicate with each other. The user equipment and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0071] User devices include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and personal computers (PCs). Smart mobile devices include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and internet-connected cars. Smart home devices include smart TVs and smart refrigerators. Wearable devices include smart watches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0072] A server can be a standalone server, a server cluster, or even a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service ecosystem. It addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0073] Before performing a video generation task, the VAE training device can use the method provided in the embodiment of the present application to train the VAE.

[0074] The user inputs control conditions (e.g., description text) through the user device, which then sends them over the network to the video generation device on the server side. The video generation device uses the trained VAE combined with the diffusion model to generate the target video and returns it to the user device over the network. The user device then displays the received target video to the user.

[0075] Apart from Figure 5 In addition to the architecture shown, a computer terminal device with strong computing power can also use the method provided in the embodiment of the present application to train VAE and / or generate target videos.

[0076] It should be understood that Figure 5 The number of user devices, VAE training devices, and video generation devices in the embodiment is merely illustrative. Any number of user devices, VAE training devices, and video generation devices may be used depending on the implementation requirements.

[0077] In order to verify the effectiveness of the VAE training method provided in the embodiment of the present application, the same validation data set was used to verify the VAE trained using the traditional method and the VAE trained using the method provided in the embodiment of the present application, including the following contents: The VAE trained using the traditional method and the VAE trained using the method provided in the embodiment of the present application cluster the same video collection. If the clustering effect is better, it means that the semantics of the VAE is stronger. Among them, the clustering effect can be measured by the ratio of the intra-cluster spacing to the inter-cluster spacing. The smaller the ratio, the better the clustering effect. After verification, the ratio corresponding to the VAE trained using the traditional method is 0.9, while the ratio corresponding to the VAE trained using the method provided in the embodiment of the present application is reduced to 0.78.

[0078] The VAE trained using the traditional method and the VAE trained using the method provided in the embodiments of the present application are combined with the diffusion model. The VAE trained using the method provided in the embodiments of the present application can increase the training convergence speed of the diffusion model by more than 2 times.

[0079] In addition, the high-frequency energy of the latent space tensor in the frequency domain output by the VAE trained using the traditional method and the VAE trained using the method provided in the embodiments of the present application was analyzed. The high-frequency energy corresponding to the VAE trained using the method provided in the embodiments of the present application was reduced by 20%, indicating that the ability to retain video generation details and suppress artifacts is stronger.

[0080] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0081] Figure 6 This is a schematic block diagram of a VAE training device provided in an embodiment of the present application, wherein the VAE includes an encoder and a decoder. Figure 6 As shown, the apparatus 600 may include: a sample acquisition unit 601 and a model training unit 602. The main functions of each component unit are as follows: The sample acquisition unit 601 is configured to acquire training data including a plurality of video samples.

[0082] The model training unit 602 is configured to train the variational autoencoder using training data; wherein the training includes: inputting video samples into the variational autoencoder and the video base model respectively, obtaining the latent space tensor output by the encoder in the variational autoencoder and the first tensor output by the video base model; mapping the latent space tensor into a second tensor, the second tensor having the same shape as the first tensor; determining the value of the first loss function term using the difference index value between the first tensor and the second tensor; determining the value of the loss function using at least the value of the first loss function term, and updating the model parameters of the variational autoencoder using the value of the loss function.

[0083] As one of the feasible ways, the above-mentioned model training unit 602 can be specifically configured as follows: obtaining the latent space tensors respectively output by multiple coding network layers in the encoder, and obtaining the first tensors respectively output by multiple coding network layers in the video basic model; mapping the latent space tensors respectively output by multiple coding network layers in the encoder to obtain multiple second tensors; determining the difference index values between the multiple second tensors and the first tensors output by the corresponding coding network layers in the video basic model, performing weighted summation on the determined multiple difference index values, and obtaining the value of the first loss function term.

[0084] As one of the feasible ways, when the above-mentioned model training unit 602 uses at least the value of the first loss function item to determine the value of the loss function, it can be specifically configured as follows: using the value of the first loss function item and the value of the second loss function item to determine the value of the loss function, wherein the value of the second loss function item is obtained by using the difference between the first restored video and the video sample, and the first restored video is obtained by decoding by the decoder using the latent space tensor.

[0085] As another feasible method, the model training unit 602 can also be configured to: perform the same affine transformation on the video sample and the latent space tensor, input the latent space tensor obtained after the affine transformation into the decoder, and obtain the second restored video decoded by the decoder.

[0086] When the model training unit 602 determines the value of the loss function by at least using the value of the first loss function item, it can be specifically configured as: determining the value of the loss function item by using the value of the first loss function item and the value of the third loss function item, wherein the value of the third loss function item is obtained by using the difference between the second restored video and the video sample obtained after affine transformation.

[0087] Furthermore, the model training unit 602 may also be configured to obtain the latent space tensor and the power spectrum of the latent space tensor after affine transformation in the frequency domain.

[0088] When the model training unit 602 uses the value of the first loss function item and the value of the third loss function item to determine the value of the loss function item, it can be specifically configured as: using the value of the first loss function item, the value of the third loss function item and the value of the fourth loss function item to determine the value of the loss function item, wherein the fourth loss function item is obtained by using the difference in high-frequency components between the power spectrum corresponding to the latent space tensor and the power spectrum corresponding to the latent space tensor after affine transformation.

[0089] Figure 7 A schematic block diagram of a video generation device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the apparatus 700 may include: a condition acquisition unit 701 and a video generation unit 702. The main functions of each component unit are as follows: The condition acquisition unit 701 is configured to acquire a control condition.

[0090] The video generation unit 702 is configured to use the encoder in the variational autoencoder to obtain the latent space tensor of the noisy video; use the control condition to denoise the latent space tensor of the noisy video to obtain the denoised latent space tensor; use the decoder of the variational autoencoder to decode the denoised latent space tensor to obtain the target video.

[0091] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or device embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0092] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0093] And an electronic device comprising: one or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0094] The present application also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods described in the aforementioned method embodiments when executed by a processor.

[0095] in, Figure 8 The electronic device architecture is shown as an example, and may include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, the video display adapter 811, the disk drive 812, the input / output interface 813, the network interface 814, and the memory 820 may be communicatively connected via a communication bus 830.

[0096] The processor 810 may be implemented as a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and may be used to execute relevant programs to implement the technical solutions provided in this application.

[0097] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store an operating system 821 for controlling the operation of the electronic device 800 and a basic input and output system (BIOS) 822 for controlling the low-level operations of the electronic device 800. In addition, a web browser 823, a data storage management system 824, and a VAE training device 600 / video generation device 700, etc. can also be stored. The above-mentioned VAE training device 600 / video generation device 700 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0098] The input / output interface 813 is used to connect to input / output modules to enable information input and output. The input / output modules can be configured as components within the device (not shown) or externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, and various sensors, while output devices may include a display, speaker, vibrator, indicator light, and the like.

[0099] The network interface 814 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

[0100] The bus 830 comprises a pathway for transmitting information between the various components of the device (eg, the processor 810 , the video display adapter 811 , the disk drive 812 , the input / output interface 813 , the network interface 814 , and the memory 820 ).

[0101] It should be noted that although the above device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0102] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer program product. The computer program product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0103] The above is a detailed introduction to the technical solutions provided by this application. Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this application.

Claims

1. A method for training a variational autoencoder, wherein the variational autoencoder comprises an encoder and a decoder; The method comprises: Obtaining training data including a plurality of video samples; The variational autoencoder is trained using the training data; wherein the training includes: inputting the video samples into the variational autoencoder and the video base model respectively, obtaining the latent space tensor output by the encoder in the variational autoencoder and the first tensor output by the video base model; mapping the latent space tensor into a second tensor having the same shape as the first tensor; determining the value of a first loss function term using the difference index value between the first tensor and the second tensor; determining the value of the loss function using at least the value of the first loss function term, and updating the model parameters of the variational autoencoder using the value of the loss function.

2. The method according to claim 1, characterized in that The obtaining of the latent space tensor output by the encoder in the variational autoencoder and the first tensor output by the video base model comprises: obtaining the latent space tensors respectively output by a plurality of encoding network layers in the encoder, and obtaining the first tensors respectively output by a plurality of encoding network layers in the video base model; Mapping the latent space tensor into a second tensor includes: mapping the latent space tensors respectively output by a plurality of encoding network layers in the encoder to obtain a plurality of second tensors; The value of the first loss function item is determined by using the difference index value between the first tensor and the second tensor, including: determining the difference index values between the multiple second tensors and the first tensor output by the corresponding encoding network layer in the video basic model, and performing weighted summation on the determined multiple difference index values to obtain the value of the first loss function item.

3. The method according to claim 1 or 2, characterized in that The determining the value of the loss function by at least using the value of the first loss function term includes: The value of the loss function is determined using the value of the first loss function item and the value of the second loss function item, wherein the value of the second loss function item is obtained by using the difference between the first restored video and the video sample, and the first restored video is obtained by the decoder decoding using the latent space tensor.

4. The method according to claim 1 or 2, characterized in that The training further includes: performing the same affine transformation on the video sample and the latent space tensor, inputting the latent space tensor obtained after the affine transformation into the decoder, and obtaining a second restored video obtained by decoding by the decoder; The method of determining the value of the loss function by at least using the value of the first loss function item includes: determining the value of the loss function item by using the value of the first loss function item and the value of the third loss function item, wherein the value of the third loss function item is obtained by using the difference between the second restored video and the video sample obtained after the affine transformation.

5. The method according to claim 4, characterized in that The training further includes: obtaining power spectra of the latent space tensor and the latent space tensor after the affine transformation in the frequency domain; The determining the value of the loss function term by using the value of the first loss function term and the value of the third loss function term includes: The value of the loss function term is determined using the value of the first loss function term, the value of the third loss function term, and the value of the fourth loss function term, wherein the fourth loss function term is obtained by using the difference in high-frequency components between the power spectrum corresponding to the latent space tensor and the power spectrum corresponding to the latent space tensor after the affine transformation.

6. A video generation method, characterized in that: The method comprises: Obtain control conditions; The encoder in the variational autoencoder is used to obtain the latent space tensor of the noisy video; Denoising the latent space tensor of the noisy video using the control condition to obtain a denoised latent space tensor; Decoding the denoised latent space tensor using a decoder of the variational autoencoder to obtain a target video; The variational autoencoder is pre-trained using the method according to any one of claims 1 to 5.

7. A training device for a variational autoencoder, the variational autoencoder comprising an encoder and a decoder; characterized in that: The device comprises: A sample acquisition unit, configured to acquire training data including a plurality of video samples; A model training unit is configured to train the variational autoencoder using the training data; wherein the training includes: inputting the video samples into the variational autoencoder and the video base model respectively, obtaining the latent space tensor output by the encoder in the variational autoencoder and the first tensor output by the video base model; mapping the latent space tensor into a second tensor, the second tensor having the same shape as the first tensor; determining the value of a first loss function term using the difference index value between the first tensor and the second tensor; determining the value of the loss function using at least the value of the first loss function term, and updating the model parameters of the variational autoencoder using the value of the loss function.

8. A video generating device, characterized in that: The device comprises: a condition acquisition unit configured to acquire a control condition; A video generation unit is configured to use an encoder in a variational autoencoder to obtain a latent space tensor of a noisy video; use the control condition to denoise the latent space tensor of the noisy video to obtain a denoised latent space tensor; use the decoder of the variational autoencoder to decode the denoised latent space tensor to obtain a target video; wherein the variational autoencoder is pre-trained using the method described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method according to any one of claims 1 to 6.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images