Video generation method and device, electronic equipment and computer storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a computer storage medium, and can be applied to the technical field of artificial intelligence. The method comprises the following steps: acquiring an input text for indicating a video generation demand and an input video serving as a video generation reference; compressing, using a video encoder, the input video into a first latent code in a latent space; de-noising the first latent code added with the random noise by taking the input video as a de-noising condition to obtain a second latent code; reconstructing the second latent code into a first video according to the input text by using a first video decoder symmetrical to the video encoder; and a sharing up-sampling module is utilized to perform up-sampling on the first video to obtain a second video, the resolution of the first video and the resolution of the input video are the same and are smaller than the resolution of the second video, and the sharing up-sampling module shares training parameters of the video generation task and the resolution optimization task.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a video generation method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and computer technology, the use of AI for video generation has become increasingly widespread. Video generation is a multimedia technology designed to generate high-definition videos, such as 4K or 8K, from text as input.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technologies: the overall generation framework of the single-stage video generation method is large and the computational complexity is high; the video generation chain of the multi-stage video generation method is long, and errors in a certain stage will gradually amplify in subsequent stages, affecting the generation quality, and the overall computational complexity is still large. Summary of the Invention

[0004] In view of this, the present disclosure provides a video generation method, apparatus, electronic device, and computer storage medium.

[0005] One aspect of this disclosure provides a video generation method, comprising: acquiring input text indicating video generation requirements and an input video serving as a reference for video generation; compressing the input video into a first latent code in a latent space using a video encoder; denoising the first latent code with added random noise using the input text as a denoising condition to obtain a second latent code; reconstructing a first video from the second latent code based on the input text using a first video decoder symmetrical to the video encoder; and upsampling the first video using a shared upsampling module to obtain a second video, wherein the first video and the input video have the same resolution but a lower resolution than the second video, and the shared upsampling module shares training parameters for both the video generation task and the resolution optimization task.

[0006] According to embodiments of this disclosure, a first video decoder symmetrical to the video encoder is used to reconstruct a second latent code into a first video based on input text. This includes: using the first video decoder to perform three-dimensional transposed convolution processing on the second latent code to increase the feature dimension of the second latent code located in the latent space to the feature dimension of the input video, thereby obtaining first intermediate information; fusing the semantic feature information of the input text and the first intermediate information to obtain fused information; and using an activation function to reconstruct the fused information into the first video.

[0007] According to an embodiment of this disclosure, a second latent code is obtained by denoising a first latent code with added random noise using the input text as the denoising condition. This includes: using the video generation requirements indicated by the input text as semantic guidance, iteratively denoising the first latent code with added random noise using a diffusion model to obtain a second latent code that semantically matches the input text.

[0008] According to embodiments of this disclosure, a video generation task is used to generate a first video, and a resolution optimization task is performed during the training process of the shared upsampling module to reconstruct a third video from a third latent code using a pre-trained second video decoder. The third latent code is obtained by compressing a first labeled video, which is a labeled video with the same resolution as the sample input video. The shared upsampling module is trained using the sample first video and the third video, and the sample first video is obtained by reconstructing the sample second latent code.

[0009] According to embodiments of this disclosure, a pre-trained second video decoder reconstructs a third video from a third latent code in the following manner: performing a three-dimensional transposed convolution on the third latent code to increase the feature dimension of the third latent code located in the latent space to the feature dimension of the first labeled video, thereby obtaining second intermediate information. The second video decoder and the first video decoder obtain the second intermediate information and the first intermediate information respectively using convolutional layers with the same structure, and the parameters of the convolutional layers of the second video decoder and the first video decoder are different; and using an activation function to reconstruct the third video from the second intermediate information.

[0010] According to embodiments of this disclosure, the shared upsampling module is trained as follows: The pre-trained shared upsampling module is used to upsample the first and third sample videos respectively to obtain the second and fourth sample videos; a target loss function is determined based on at least two of the sample input video, the second and fourth sample videos, the first and second label videos, and the first and third sample videos, wherein the second label video has the same resolution as the second sample video, and the second label video is downsampled to obtain the first label video; and under the constraint of the target loss function, the pre-trained shared upsampling module, the pre-trained video encoder, the pre-trained first video decoder, and the pre-trained second video decoder are jointly fine-tuned until the target loss function converges, resulting in the shared upsampling module, the video encoder, the first video decoder, and the second video decoder.

[0011] According to embodiments of this disclosure, the target loss function includes: a video generation task loss and a resolution optimization task loss; the video generation task loss includes at least one of the following: a first loss for indicating pixel reconstruction differences between a sample second video and a second tag video; a second loss for indicating pixel reconstruction differences between a sample first video and a first tag video; a third loss for indicating image semantic differences between a sample second video and a second tag video; the resolution optimization task loss includes at least one of the following: a fourth loss for indicating pixel reconstruction differences between a fourth video and a second tag video; a fifth loss for indicating pixel reconstruction differences between a third video and a first tag video; a sixth loss for indicating image semantic differences between a fourth video and a second tag video; and a seventh loss for indicating pixel differences between adjacent video frames in the fourth video.

[0012] According to embodiments of this disclosure, the video encoder shares training parameters for both the video generation task and the resolution optimization task: the target loss function further includes a shared constraint loss for indicating the distribution differences between the third latent code and the fourth latent code in the latent space, wherein the fourth latent code is obtained by compressing the sample first video using a pre-trained video encoder; the video generation task loss further includes a regularization loss for the pre-trained video encoder.

[0013] Another aspect of this disclosure provides a video generation apparatus, comprising: an acquisition module for acquiring input text indicating video generation requirements and an input video as a reference for video generation; a compression module for compressing the input video into a first latent code in a latent space using a video encoder; a denoising module for denoising the first latent code with added random noise using the input text as a denoising condition to obtain a second latent code; a reconstruction module for reconstructing the second latent code into a first video based on the input text using a first video decoder symmetrical to the video encoder; and a generation module for upsampling the first video using a shared upsampling module to obtain a second video, wherein the first video and the input video have the same resolution but a resolution smaller than that of the second video, and the shared upsampling module shares training parameters for both the video generation task and the resolution optimization task.

[0014] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the methods described above.

[0015] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0016] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0017] In the embodiments of this disclosure, by employing an architecture of a video encoder, denoising, and a first video decoder, the input video can be compressed into a low-dimensional latent space for denoising to generate a second latent code matching the input text and its corresponding high-dimensional first video, significantly reducing the computational complexity of video generation. Furthermore, since the shared upsampling module shares the parameters of the video generation task and the resolution upsampling task, while using the simple shared upsampling module to upsample the first video, semantic recovery and detail recovery performance can be balanced, improving the resolution of the generated second video. Attached Figure Description

[0018] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture for video generation can be applied according to embodiments of this disclosure is illustrated.

[0020] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 3A The schematic diagram illustrates the structure of generating a second video according to an embodiment of the present disclosure.

[0022] Figure 3B The diagram illustrates a structural schematic of a first video decoder generating a second video according to a specific embodiment of the present disclosure.

[0023] Figure 4 The illustration shows a scenario diagram of the training phase according to an embodiment of the present disclosure.

[0024] Figure 5 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 6 A block diagram of an electronic device suitable for implementing a video generation method and apparatus according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the embodiments of this disclosure, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and other security. In the embodiments of this disclosure, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0031] For example, single-stage video generation can use end-to-end video models. However, in super-resolution (SR) video generation scenarios, 4K video (3840×2160) has a huge spatiotemporal volume, requiring significant hardware costs during training and substantial GPU memory usage during inference. Taking 29 frames as an example, the original pixel count of a high-definition video is approximately 29×3840×2160×3≈215 million pixels. With each additional video frame, the computational complexity of the model increases cubically.

[0032] To efficiently generate high-fidelity, spatiotemporally consistent, and detail-rich long-term videos with limited computing resources, directly extending traditional generative models to ultra-high definition scales can lead to problems such as memory overload, unstable training, and blurred details. Therefore, a multi-stage video generation approach can be adopted.

[0033] Multi-stage video generation can be implemented using a two-stage cascaded generation framework. In the first stage, a latent space diffusion model (such as Stable VideoDiffusion) can be used to generate medium-resolution videos (typically 576×1024 or 720p). In the second stage, a dedicated super-resolution model is used to upsample the results to 4K / 8K and enhance high-frequency details and spatiotemporal consistency. For example, the super-resolution model could be a Swin Transformer structure or a Video Super-Resolution (VSR) network based on a 3D Convolutional Neural Network (CNN). In this example, if the video generated in the first stage has structural errors such as distorted faces or flickering objects, the super-resolution task in the second stage will amplify these defects, even introducing false textures and edge ringing, leading to compounded errors. Furthermore, the sequential execution of forward inference in the two stages gradually increases video generation latency and engineering complexity, such as the involvement of intermediate data caching and transformation, which increases data processing and training costs.

[0034] To address this issue, embodiments of this disclosure propose a video generation method to solve the technical problem of high computational complexity in the video generation process.

[0035] Figure 1 The illustration schematically depicts an exemplary system architecture for video generation that can be applied according to embodiments of this disclosure. For example... Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software (for example only). The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and supporting web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers, etc.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the video generation method provided in this embodiment can generally be executed by server 105 or a server cluster, and correspondingly, the video generation device can generally be set in server 105 or a server cluster. Alternatively, the video generation method provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or it can be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103, and correspondingly, the video generation device provided in this embodiment can also be set in the aforementioned terminal devices. Furthermore, the video generation method provided in this embodiment can also be executed by a system composed of terminal devices and servers, and correspondingly, the video generation device can also be set in the terminal devices and the server.

[0039] For example, a user can interact with the first terminal device 101, the second terminal device 102, or the third terminal device 103 to input the aforementioned input text and input video. Correspondingly, the server 105 obtains the input text and input video through the aforementioned terminal devices, executes the video generation method, and feeds back the generated second video to the user through the aforementioned terminal devices.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] Figure 2 A flowchart illustrating a video generation method according to an embodiment of this disclosure is shown schematically. Figure 2 As shown, the method includes operations S210~S250.

[0042] In operation S210, input text indicating the video generation requirements and input video as the basis for video generation are obtained.

[0043] When there is a need for video generation, users can input a piece of text and a piece of video. The text is used to indicate the video generation requirement and can be called the input text; the video is used as the basis for video generation and can be called the input video.

[0044] Video generation requirements can be varied, including video content, objects within the video, style, dialogue, movement, and theme. For example, the input text could be: "Generate a beautiful video of waves crashing against rocks at sunset on the beach, with a warm color tone and a slight slow-motion effect."

[0045] The input video typically serves as the baseline for video generation. Its content, objects, and style can all be used as reference points. Information that differs from the baseline can be replaced with content that aligns with the video generation requirements. For example, if the input video is a retro-style animation of dialogue, and the user expects the generated video to maintain this hand-drawn feel and color scheme, the input text could be: "Generate a beautiful video of waves crashing against rocks at sunset on the beach, following the style of the input video. The video should have a warm color tone and a slight slow-motion effect." The content of the final output video (second video) can be unrelated to the dialogue in the input video.

[0046] In operation of S220, the input video is compressed into the first latent code in the latent space using a video encoder.

[0047] A video encoder is used to compress each frame of an input video into a low-dimensional latent space in time and / or space to obtain a first latent code. The low-dimensional latent space is an abstract vector space defined by a mathematical model, with dimensions much lower than the input video. The video encoder is used to reduce the dimensionality and extract features from the high-dimensional data of the input video to obtain the first latent code, also known as the latent representation. The first latent code is an encoded representation of key features in the input video (such as semantic or motion information like object shape, texture, color distribution, and scene layout).

[0048] A video encoder can be a coding network capable of dimensionality reduction and feature extraction. For example, a video encoder can be the encoder in a variational autoencoder (VAE). It's important to note that if the video encoder is a VAE encoder, then the first video decoder, which is symmetrical to the video encoder in the following text, can be the decoder of the VAE. Symmetry refers to the fact that the video encoder compresses the video into the inverse transform of the latent code, that is, it remaps the latent code back to the video; the two operations are completely opposite. For example, if the video encoder performs N compressions, then the first video decoder performs N inverse transforms.

[0049] In operation S230, the input text is used as the denoising condition to denoise the first latent code with added random noise, thereby obtaining the second latent code.

[0050] The first latent code is a latent representation based on the input video. After adding random noise, the first latent code can be regarded as a latent code containing more information. Then, based on the first latent code with added random noise, noise can be gradually removed to obtain a brand new second latent code that matches the input text.

[0051] In embodiments of this disclosure, denoising specifically refers to the targeted denoising of a first latent code with added random noise multiple times under the guidance of denoising conditions until a second latent code matching the input text is obtained. Since the input text indicates the user's video generation requirements, using the input text as a denoising condition allows the denoising process to proceed in the direction of generating a video latent code that conforms to the description of the input text, ultimately resulting in a second latent code that is highly consistent with the user's video generation requirements.

[0052] In this embodiment, guided by denoising conditions, multiple iterations of denoising can be performed using a denoising network to obtain the final second latent code. The denoising network can be a diffusion transformer (DiT), a latent diffusion model (LDM), or the like.

[0053] In operation S240, using a first video decoder that is symmetrical to the video encoder, the second latent code is reconstructed into the first video based on the input text.

[0054] In this embodiment, the input video is compressed into a low-dimensional latent space by a video encoder to obtain a first latent code. Then, after denoising the latent space to obtain a second latent code that matches the video generation requirements, the second latent code can be inversely transformed to recover the high-dimensional first video. It is understood that this first video has the same resolution as the input video.

[0055] Although the latent space denoising process is guided by the input text and the second latent code matches the video generation requirements, the dimensional transformation of the low-dimensional latent space may also have defects, or the second latent code obtained by the denoising process may also have defects. Therefore, when reconstructing the low-dimensional second latent code into a high-dimensional first video, the input text is still introduced to assist the first video decoder in accurately understanding the semantic information contained in the second latent code and correctly mapping it back to the high-dimensional first video.

[0056] As mentioned above, the first video decoder can be symmetrical to the video encoder, such as the first video decoder being a VAEdecoder.

[0057] In operation S250, the shared upsampling module is used to upsample the first video to obtain the second video. The first video and the input video have the same resolution, but the resolution of the second video is smaller. The shared upsampling module shares the training parameters of the video generation task and the resolution optimization task.

[0058] In operations S210 to S240, a first video is obtained after denoising and dimensionality upscaling in the low-dimensional latent space. The content of the first video matches the video generation requirements, but its resolution is the same as the input video, which does not yet meet the user's resolution requirements. Therefore, based on the above video generation architecture, a shared upsampling module is added, which upsamples the first video for the user.

[0059] For example, the resolution of the second video can be 4K, 8K, etc., as long as it is higher than the resolution of the input video and the first video.

[0060] The shared upsampling module can be implemented using a simple network structure incorporating an attention mechanism, ensuring that the learned training parameters can reconstruct a high-resolution second video from a low-resolution first video. The training parameters can be understood as the mapping relationship between the high-resolution video (second video) and the low-resolution video (first video).

[0061] The video generation task is the same as the process of generating the first video mentioned above, and the resolution optimization task can be the task of increasing the resolution. It should be noted that the above framework does not perform the resolution optimization task during the inference and generation of the second video. It only performs the two tasks separately during the training process using a shared upsampling module with the same set of training parameters. That is, the resolution optimization task is only used to assist in optimizing the shared upsampling module during the training process.

[0062] In this embodiment, considering that the shared upsampling module adopts a simpler model architecture compared to the complete two-stage cascaded model architecture, its improvement in resolution is limited. Both video generation and resolution optimization tasks essentially involve semantic understanding and detail generation and restoration of video content (video generation focuses on semantic understanding of video content, while resolution optimization focuses on improving video resolution). Therefore, by leveraging the complementary characteristics of the two tasks to train the shared upsampling module, the trained module can simultaneously handle semantic restoration and detail restoration during the upsampling of the first video, thereby improving resolution performance.

[0063] In the embodiments of this disclosure, by employing an architecture of a video encoder, denoising, and a first video decoder, the input video can be compressed into a low-dimensional latent space for denoising to generate a second latent code matching the input text and its corresponding high-dimensional first video, significantly reducing the computational complexity of video generation. Furthermore, since the shared upsampling module shares the parameters of the video generation task and the resolution upsampling task, while using the simple shared upsampling module to upsample the first video, semantic recovery and detail recovery performance can be balanced, improving the resolution of the generated second video.

[0064] According to an embodiment of this disclosure, a second latent code is obtained by denoising a first latent code with added random noise using the input text as the denoising condition. This includes: using the video generation requirements indicated by the input text as semantic guidance, iteratively denoising the first latent code with added random noise using a diffusion model to obtain a second latent code that semantically matches the input text.

[0065] The Diffusion Transformer (DiT) model can generate videos based solely on text descriptions, or it can generate modules tailored to user video generation needs based on both text and video. The Diffusion model can obtain a second latent code through iterative denoising.

[0066] For example, the diffusion model can perform iterative denoising through multiple time steps. In the (i+1)th time step, the input text is used as the denoising condition, and the intermediate latent code obtained in the i-th time step is denoised in a targeted manner to obtain the intermediate latent code in the (i+1)th time step. After iterating through a predetermined number of time steps, the intermediate latent code obtained in the last time step is the second latent code.

[0067] The above denoising process is similar to plain text-based denoising, except that the initial latent code is transformed from random noise to a first latent code after adding random noise. The random noise in this embodiment can be Gaussian noise.

[0068] To meet the video generation requirements indicated by the input text, a text encoder can be used to encode the input text, resulting in semantic feature information (text embedding) with rich semantic information.

[0069] For example, in the (i+1)th time step, the diffusion model can fuse semantic feature information with the intermediate latent code from the ith time step, predict the noise distribution based on the fused features, and then remove the predicted noise distribution from the intermediate latent code of the ith time step to obtain the intermediate latent code for the (i+1)th time step. In each time step, the video generation requirements of the input text are fused into the intermediate latent code, ensuring that the second latent code obtained in the last time step can semantically match the input text.

[0070] In the embodiments of this disclosure, the input video is used as the generation basis, and a second latent code that semantically matches the input text is generated under the semantic guidance of the input text using a diffusion model. This ensures that the video content of the subsequent first and second videos can meet the user's video generation requirements and guarantee the accuracy of the video content.

[0071] Figure 3A A schematic diagram illustrating the structure for generating a second video according to an embodiment of this disclosure is shown. Figure 3A As shown, the user's input video and input text can be input into the video encoder 301 and text encoder 303, respectively, to obtain the first latent code and semantic feature information. Then, the first latent code and semantic feature information can be used as input to the diffusion model 302. Under the influence of the semantic feature information, the diffusion model 302 iteratively denoises the first latent code with added random noise to obtain the second latent code. Next, using the first video decoder 304, which is symmetrical to the video encoder 301, the second latent code in the low-dimensional latent space is reconstructed into the first video based on the semantic feature information. Then, the shared upsampling module 305 upsamples the first video to obtain the second video.

[0072] In one embodiment, if the dimensions of the input video are 64 (height H) × 64 (width W) × 16 (frames T), the video encoder can downsample the input video by 8 times. The dimensions of the compressed first latent code are 32 × 32 × 4 × C (C is the number of channels, corresponding to the design of the video encoder), that is, spatial downsampling is performed by 2 times in space (64 × 64) and temporal downsampling is performed by 4 times in time (16). The dimensions of the second latent code obtained after denoising are the same as those of the first latent code. The dimensions of the first video reconstructed by the first video decoder after the second latent code can be 64 × 64 × 16. The dimensions of the second video obtained by the shared upsampling module can be (256 × 256 × 16), that is, the spatial resolution is improved by 4 times.

[0073] In one embodiment, the video encoder can use 3D convolution and a spatiotemporal attention mechanism to compress the input video into a first latent code. For example, the video encoder can first downsample each video frame of the input video in time and space (e.g., 4x spatial downsampling and 2x temporal downsampling) through multiple 3D convolutional layers to reduce the dimensionality of the input video; then, it can extract features from the downsampled feature information through a spatiotemporal attention mechanism to obtain the aforementioned first latent code.

[0074] According to embodiments of this disclosure, a first video decoder symmetrical to the video encoder is used to reconstruct a second latent code into a first video based on input text. This includes: using the first video decoder to perform three-dimensional transposed convolution processing on the second latent code to increase the feature dimension of the second latent code located in the latent space to the feature dimension of the input video, thereby obtaining first intermediate information; fusing the semantic feature information of the input text and the first intermediate information to obtain fused information; and using an activation function to reconstruct the fused information into the first video.

[0075] For example, the first video decoder may include a convolutional layer, a fusion layer, and an output layer, which are used to obtain the first intermediate information, the fusion information, and the first video, respectively.

[0076] Figure 3B This schematically illustrates a structural diagram of a first video decoder generating a second video according to a specific embodiment of the present disclosure. Figure 3A Their overall structures are similar. Figure 3B Under the influence of semantic feature information, the diffusion model 302 iteratively denoises the first latent code with added random noise to obtain the second latent code. The second latent code is then passed through the convolutional layer, fusion layer (the semantic feature information also passes through the fusion layer), and output layer in the first video decoder 304 to obtain the first video. After that, the first video is upsampled to the second video by the shared upsampling module 305.

[0077] The convolutional layer of the first video decoder can perform inverse temporal and spatial upsampling (e.g., 2x temporal upsampling and 4x spatial upsampling) through 3D transposed convolution operations to obtain the first intermediate information. Symmetrical to the 3D convolution processing in the video encoder, the 3D transposed convolution operation in the first video decoder can effectively recover the detailed features of the video data in both temporal and spatial dimensions, maintaining parameter symmetry with the corresponding 3D convolution in the video encoder.

[0078] When the first video decoder performs the conversion from latent code to video, if it only has the latent code without the original input text as a reference, the reconstructed video may have deviations or ambiguities in terms of detail depiction and semantic consistency maintenance, resulting in poor continuity of the video across time and space. Therefore, after performing 3D transposed convolution in the convolutional layer, the semantic feature information obtained by the text encoder from encoding the input text is reused for semantic guidance. Unlike the diffusion model, the fusion layer of the first video decoder can achieve the fusion of the semantics of the input text and the second latent code at the feature level.

[0079] In one embodiment, semantic feature information and first intermediate information can be fused using Adaptive Group Normalization (AdaGN). For example, semantic feature information can be mapped to two auxiliary features using a multilayer perceptron and activation layers; the two auxiliary features are then applied to the scaling (i.e., scale) and offset of the normalized first intermediate information to obtain the fused information.

[0080] In another embodiment, semantic feature information and first intermediate information can be fused through a cross-attention mechanism. For example, the first intermediate information can be used as a query, and the semantic feature information as a key and value. The attention weight is obtained by calculating the similarity between the query and the key, and then the value is weighted and summed using this weight, so that the first intermediate information can dynamically focus on the text semantic content in the semantic feature information that is relevant to the current decoding stage.

[0081] The fusion information obtained from the activation layer can be mapped to the value (0~1) of each pixel in each video frame using a non-linear mapping of the activation function. Then, the pixel value of each pixel in each video frame can be determined by using the relationship between the baseline pixel value (0~255) and the mapped value (0~1). Activation functions can include ReLU, Sigmoid, or Tanh, etc.

[0082] In the embodiments of this disclosure, during the process of reconstructing the first video from the second latent code using the first video decoder, the mapping between the low-dimensional latent space and the high-dimensional video space is performed through semantic guidance of the input text. This effectively avoids content deviations or semantic ambiguities caused by the lack of textual semantic guidance, improving the continuity and semantic consistency of the video across time and space. Furthermore, this semantic guidance method can also correct potential defects in the second latent code generated by the diffusion model, or avoid amplifying such defects, ensuring the quality of the generated second video.

[0083] According to embodiments of this disclosure, a video generation task is used to generate a first video, and a resolution optimization task is performed during the training process of the shared upsampling module to reconstruct a third video from a third latent code using a pre-trained second video decoder. The third latent code is obtained by compressing a first labeled video, which is a labeled video with the same resolution as the sample input video. The shared upsampling module is trained using the sample first video and the third video, and the sample first video is obtained by reconstructing the sample second latent code.

[0084] As stated above, the video generation task can be viewed as a process of reconstructing the video using the second latent code using the first video decoder, and it is applied during the training and inference phases; the resolution optimization task can be viewed as a process of reconstructing the video using the pre-trained second video decoder, and it is only applied during the training phase. For example, the first and second video decoders can be regarded as task-specific decoders, and can be called the generation decoder and the super-resolution decoder (SR decoder), respectively.

[0085] In one embodiment, during training, the pre-trained second video decoder can be used directly without fine-tuning; alternatively, the pre-trained second video decoder can be fine-tuned together with the inference architecture (such as the video encoder, diffusion model, first video decoder, and shared upsampling module mentioned above) as a multi-task architecture to further optimize the inference architecture using the optimized second video decoder (i.e., the fine-tuned pre-trained second video decoder).

[0086] Understandably, the information in the video generation task during the inference phase corresponds to that in the training phase. For example, the input video, input text, first latent code, second latent code, first video, and second video correspond to the sample input video, sample input text, sample first latent code, sample second latent code, sample first video, and sample second video, respectively. For instance, using the sample input text as a denoising condition, the sample first latent code with added random noise is denoised to obtain the sample second latent code. The sample first latent code is obtained by compressing the sample input video using the video encoder. The pre-trained first video decoder reconstructs the sample first video from the sample second latent code, and then uses the pre-trained shared upsampling module to upsample the sample first video to the sample second video.

[0087] Unlike the inference phase, the training phase also includes a second video decoder, whose input is a third latent code obtained by compressing the first-labeled video, and whose output is a third video. Similar to video generation, the pre-trained shared upsampling module can also upsample the third video.

[0088] During the training phase, each sample input text and sample input video uniquely corresponds to a labeled video that has been upgraded in resolution and meets the semantic requirements of the sample input text. In one embodiment, this labeled video is the second labeled video; to ensure that the shared upsampling module has high resolution enhancement performance, the first labeled video is obtained by downsampling the second labeled video.

[0089] During the training phase described above, the input to the pre-trained shared upsampling module consists of videos from two sources: the video generation task and the resolution optimization task. The pre-trained shared upsampling module can learn upsampling features and mapping relationships applicable to video inputs from different sources. Thus, during the inference phase, it can leverage the advantages of the resolution optimization task to perform video mapping for the video generation task. This not only generates videos that match the user's video generation needs but also effectively improves the video resolution.

[0090] According to embodiments of this disclosure, a pre-trained second video decoder reconstructs a third video from a third latent code in the following manner: performing a three-dimensional transposed convolution on the third latent code to increase the feature dimension of the third latent code located in the latent space to the feature dimension of the first labeled video, thereby obtaining second intermediate information. The second video decoder and the first video decoder obtain the second intermediate information and the first intermediate information respectively using convolutional layers with the same structure, and the parameters of the convolutional layers of the second video decoder and the first video decoder are different; and using an activation function to reconstruct the third video from the second intermediate information.

[0091] The difference between the pre-trained second video decoder and the first video decoder is that the first video decoder focuses on semantically guided video reconstruction, while the second video decoder only focuses on video reconstruction. Therefore, both include structurally identical video processing parts, such as the identical structure of the convolutional layers mentioned above. However, the two tasks have different focuses, and they are trained separately; therefore, the parameters of the trained convolutional layers are different.

[0092] For example, the parameters of the convolutional layers of the first video decoder focus more on capturing the relationship between semantics and video content, while the parameters of the convolutional layers of the pre-trained second video decoder focus on pixel-level detail recovery and temporal coherence.

[0093] Similar to how the convolutional layers of the first video decoder generate the first intermediate information, the pre-trained second video decoder uses convolutional layers with the same structure to generate the second intermediate information; similarly, the activation function also maps the second intermediate information to the third video, the only difference between the two is the input latent code, which will not be elaborated here.

[0094] In embodiments of this disclosure, a pre-trained second video decoder, with a structure similar to the first video decoder, is used to perform a resolution optimization task. This allows the input to the shared upsampling module to include videos from two different task sources, each with its own advantages. Therefore, the pre-trained shared upsampling module is trained using these two source videos, enabling it to better understand the semantic and detail-level requirements of the video, thereby generating a video that balances content accuracy and high resolution.

[0095] According to embodiments of this disclosure, the shared upsampling module is trained as follows: The pre-trained shared upsampling module is used to upsample the first and third sample videos respectively to obtain the second and fourth sample videos; a target loss function is determined based on at least two of the sample input video, the second and fourth sample videos, the first and second label videos, and the first and third sample videos, wherein the second label video has the same resolution as the second sample video, and the second label video is downsampled to obtain the first label video; and under the constraint of the target loss function, the pre-trained shared upsampling module, the pre-trained video encoder, the pre-trained first video decoder, and the pre-trained second video decoder are jointly fine-tuned until the target loss function converges, resulting in the shared upsampling module, the video encoder, the first video decoder, and the second video decoder.

[0096] The first-labeled video and the second-labeled video can be regarded as low-definition sample video and high-definition sample video, respectively.

[0097] Figure 4 The illustration shows a scenario diagram of the training phase according to an embodiment of the present disclosure. Figure 4As shown, the pre-trained video encoder 401, the pre-trained first video decoder 404, and the pre-trained shared upsampling module 405 are... Figure 3A The mid-video encoder, first video decoder, and shared upsampling module are similar and can all be considered as corresponding structures before training. The diffusion model 402 and text encoder 403 can be compared with... Figure 3A The diffusion model in the code is the same as that of the text encoder, meaning it is not adjusted during the training phase. The pre-trained second video decoder 406 added during the training phase is also fine-tuned along with the aforementioned structure that needs to be trained.

[0098] The newly added resolution optimization task during the training phase utilizes a pre-trained video encoder 401 to map the new input (the first labeled video) into the latent space, and then uses a pre-trained second video decoder 406 to inversely map the third latent code back into the high-dimensional video space to obtain the third video. The entire phase can be viewed as a performance evaluation of the pre-trained video encoder 401 and the pre-trained second video decoder 406.

[0099] For example, to facilitate understanding of the resolution and dimensionality changes across the entire framework, the following explanation will focus solely on dimensionality changes. The sample input video and the first label video both have dimensions of 64×64×16, and the compressed sample first and third latent codes both have dimensions of 32×32×4×C. The sample second latent code obtained by denoising the sample first latent code also has a dimension of 32×32×4×C, but they contain different information. The pre-trained first and second video decoders reconstruct the 32×32×4×C sample second and third latent codes into 64×64×16×3 (3 being RGB channels) sample first and third videos, respectively. Thus, the two video sources in the pre-trained shared upsampling module have the same dimension, and through upsampling by a factor of 2n / 4n, sample second and fourth videos with the same dimension are obtained, such as 256×256×16×3 or higher resolution.

[0100] During joint fine-tuning, to constrain each structure (especially the shared upsampling module), a target loss function can be constructed based on the input and output of the shared upsampling module, and / or the input and output of the entire training structure, and / or the output of the entire training structure and the label output. Target loss function convergence refers to the value of the target loss function obtained after multiple rounds of fine-tuning fluctuating within a preset, relatively small threshold range. When the target loss function converges, it indicates that the model's parameter adjustments have reached a relatively optimized state, capable of better fitting the patterns inherent in the sample data, and the model's generalization ability is guaranteed to a certain extent. At this point, the parameters of each pre-trained module can be considered as fine-tuned parameters, i.e., the parameters of each module are obtained.

[0101] In one specific embodiment, the shared upsampling module may be a lightweight version of the Basic Video Super-Resolution network (BasicVSR-lite) or a lightweight version of the Enhanced Deformable Video Restoration network (EDVR-lite).

[0102] The various modules of the above training framework can be trained using a multi-stage training strategy. For example, the initial shared upsampling module can be pre-trained using super-resolution data (such as the REDS dataset) through Low-Rank Adaptation (LoRA) or by introducing an adapter, resulting in the pre-trained shared upsampling module. Alternatively, after pre-training the initial shared upsampling module using super-resolution data, an intermediate shared upsampling module can be obtained; then, the intermediate shared upsampling module, along with the initial first video decoder and the initial second video decoder, can be trained separately as two task branches to obtain the pre-trained first video decoder, the pre-trained second video decoder, and the pre-trained shared upsampling module. This process can also simultaneously train the initial video encoder to obtain a pre-trained video encoder.

[0103] In the embodiments of this disclosure, by adding a pre-trained second video decoder during the training phase, and using the sample input video, sample second video, fourth video, first label video, second label video, sample first video and third video obtained under the training framework, the various modules of the training framework are jointly trained, so that each module, especially the pre-trained shared upsampling module, can better understand the needs of the video at the semantic and detail levels, thereby generating a video that balances content accuracy and high resolution.

[0104] According to embodiments of this disclosure, the target loss function includes: a video generation task loss and a resolution optimization task loss; the video generation task loss includes at least one of the following: a first loss for indicating pixel reconstruction differences between a sample second video and a second tag video; a second loss for indicating pixel reconstruction differences between a sample first video and a first tag video; a third loss for indicating image semantic differences between a sample second video and a second tag video; the resolution optimization task loss includes at least one of the following: a fourth loss for indicating pixel reconstruction differences between a fourth video and a second tag video; a fifth loss for indicating pixel reconstruction differences between a third video and a first tag video; a sixth loss for indicating image semantic differences between a fourth video and a second tag video; and a seventh loss for indicating pixel differences between adjacent video frames in the fourth video.

[0105] For example, the first loss See the following formula (1):

[0106] (1)

[0107] in, and These are the second video and the second-labeled video of the sample, respectively. It represents the L1 distance between corresponding pixel values ​​at the same position in the second video sample and the second labeled video.

[0108] For example, the second loss See formula (2) below:

[0109] (2)

[0110] in, and These are the first video and the first-label video of the sample, respectively. This represents the L1 distance between corresponding pixel values ​​at the same position in the first sample video and the first labeled video. The first loss and the second loss can be used to constrain the pixel reconstruction differences in high-resolution videos and low-resolution videos, respectively.

[0111] Regarding the third loss The semantics of the second video and the second-labeled video can be extracted using a large language model or other models for semantic extraction, resulting in a first semantic vector and a second semantic vector. The third loss can be the cosine similarity between the first and second semantic vectors. This third loss can also be called the semantic-aware loss for the video generation task.

[0112] Similarly, regarding the sixth loss The fourth video can be used to extract semantics, resulting in the third semantic vector. The sixth loss can be the cosine similarity between the third and second semantic vectors. The sixth loss can also be called the semantic perception loss for the resolution optimization task.

[0113] For example, the fourth loss See formula (3) below:

[0114] (3)

[0115] in, and These are the fourth video and the second tagged video, respectively. This represents the L1 distance between corresponding pixel values ​​at the same position in the fourth video and the second-labeled video.

[0116] For example, the fifth loss See formula (4) below:

[0117] (4)

[0118] in, and The input video for the third video and resolution optimization task (in this embodiment, the first-labeled video, i.e.) and ), This represents the L1 distance between corresponding pixel values ​​at the same position in the first video and the first labeled video of the sample.

[0119] Regarding the seventh loss, in order to constrain the temporal consistency between video frames in the video, for example, the L1 distance between corresponding pixel values ​​at the same position in two adjacent video frames in the fourth video can be calculated. The seventh loss can be obtained by summing the L1 distances between corresponding pixel values ​​at the same position in multiple adjacent video frames in the fourth video and then averaging them.

[0120] In the embodiments of this disclosure, since the training phase includes parallel video generation tasks and resolution optimization tasks, and both share the same shared upsampling module, the embodiments of this disclosure can construct loss terms for the video generation task and the resolution optimization task respectively, and fine-tune the entire training architecture under the constraints of both, so as to ensure that the shared upsampling module can learn well the detailed optimization and resolution improvement performance of the resolution optimization task, which is convenient for obtaining a high-resolution second video that matches the video generation requirements and has clear details in the inference phase.

[0121] In embodiments of this disclosure, if the video generation task loss (or resolution optimization task loss) includes at least two losses, then the at least two losses can be summed to obtain the overall video generation task loss (or resolution optimization task loss).

[0122] According to embodiments of this disclosure, the video encoder shares training parameters for both the video generation task and the resolution optimization task. The target loss function further includes a shared constraint loss to indicate the distribution differences between the third latent code and the fourth latent code in the latent space, wherein the third latent code and the fourth latent code are obtained by compressing the sample first video and the first labeled video, respectively, using a pre-trained video encoder. The video generation task loss further includes a regularization loss for the pre-trained video encoder.

[0123] like Figure 4It can be seen that, in addition to reusing the shared upsampling module, the two tasks also reuse the video encoder during the training phase. Therefore, during the joint fine-tuning process, a shared constraint loss can be added to align semantic and motion information between the two tasks in the latent space, thereby reducing the distribution difference between the two video sources in the subsequent shared upsampling module.

[0124] For example, shared constraint loss It can be determined by the following formula (5):

[0125] (5)

[0126] in, and These are the third and fourth latent codes, respectively. This represents the Euclidean distance between the third and fourth latent codes in the latent space.

[0127] For example, the regularization loss for a pre-trained video encoder. It can be determined by the following formula (6):

[0128] (6)

[0129] in, This represents the parameters of the video encoder. This indicates that the sample latent code of the first video output by the video encoder approximates a standard normal distribution. KL() represents the KL divergence calculation.

[0130] In one specific embodiment, the video generation task loss It can be represented by the following formula (7):

[0131] (7).

[0132] Resolution optimization task loss It can be represented by the following formula (8):

[0133] (8).

[0134] Target loss function It can be represented by the following formula (9):

[0135] (9)

[0136] in, , and The coefficients for each loss term can be determined based on the actual situation.

[0137] In the embodiments of this disclosure, by constraining the encoding features of the video encoder in the latent space, the video encoder can align the semantic and motion information of the two tasks in the latent space. Thus, the encoding performance can be improved by learning the information without introducing a resolution optimization task during the inference stage, thereby improving the resolution enhancement performance of the subsequent shared upsampling module.

[0138] Figure 5 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically. Figure 5 As shown, the video generation device 500 includes:

[0139] The acquisition module 510 is used to acquire input text indicating the video generation requirements and input video as the reference for video generation; the compression module 520 is used to compress the input video into a first latent code in the latent space using a video encoder; the denoising module 530 is used to denoise the first latent code with added random noise using the input text as the denoising condition to obtain a second latent code; the reconstruction module 540 is used to reconstruct the first video from the second latent code based on the input text using a first video decoder symmetrical to the video encoder; and the generation module 550 is used to upsample the first video using a shared upsampling module to obtain a second video, wherein the resolution of the first video and the input video is the same but smaller than the resolution of the second video, and the shared upsampling module shares the training parameters of the video generation task and the resolution optimization task.

[0140] According to an embodiment of this disclosure, the reconstruction module 540 includes: a convolution unit, used to perform three-dimensional transposed convolution processing on the second latent code using a first video decoder, so as to improve the feature dimension of the second latent code located in the latent space to the feature dimension of the input video, thereby obtaining first intermediate information; a fusion unit, used to fuse the semantic feature information of the input text and the first intermediate information to obtain fused information; and a reconstruction unit, used to reconstruct the fused information into a first video using an activation function.

[0141] According to an embodiment of this disclosure, the denoising module 530 includes: a denoising unit, configured to take the video generation requirements indicated by the input text as semantic guidance, and use a diffusion model to iteratively denoise a first latent code with added random noise to obtain a second latent code that semantically matches the input text.

[0142] According to embodiments of this disclosure, a video generation task is used to generate a first video, and a resolution optimization task is performed during the training process of the shared upsampling module to reconstruct a third video from a third latent code using a pre-trained second video decoder. The third latent code is obtained by compressing a first labeled video, which is a labeled video with the same resolution as the sample input video. The shared upsampling module is trained using the sample first video and the third video, and the sample first video is obtained by reconstructing the sample second latent code.

[0143] According to embodiments of this disclosure, a pre-trained second video decoder reconstructs a third video from a third latent code in the following manner: performing a three-dimensional transposed convolution on the third latent code to increase the feature dimension of the third latent code located in the latent space to the feature dimension of the first labeled video, thereby obtaining second intermediate information. The second video decoder and the first video decoder obtain the second intermediate information and the first intermediate information respectively using convolutional layers with the same structure, and the parameters of the convolutional layers of the second video decoder and the first video decoder are different; and using an activation function to reconstruct the third video from the second intermediate information.

[0144] According to embodiments of this disclosure, the shared upsampling module is trained as follows: The pre-trained shared upsampling module is used to upsample the first and third sample videos respectively to obtain the second and fourth sample videos; a target loss function is determined based on at least two of the sample input video, the second and fourth sample videos, the first and second label videos, and the first and third sample videos, wherein the second label video has the same resolution as the second sample video, and the second label video is downsampled to obtain the first label video; and under the constraint of the target loss function, the pre-trained shared upsampling module, the pre-trained video encoder, the pre-trained first video decoder, and the pre-trained second video decoder are jointly fine-tuned until the target loss function converges, resulting in the shared upsampling module, the video encoder, the first video decoder, and the second video decoder.

[0145] According to embodiments of this disclosure, the target loss function includes: a video generation task loss and a resolution optimization task loss; the video generation task loss includes at least one of the following: a first loss for indicating pixel reconstruction differences between a sample second video and a second tag video; a second loss for indicating pixel reconstruction differences between a sample first video and a first tag video; a third loss for indicating image semantic differences between a sample second video and a second tag video; the resolution optimization task loss includes at least one of the following: a fourth loss for indicating pixel reconstruction differences between a fourth video and a second tag video; a fifth loss for indicating pixel reconstruction differences between a third video and a first tag video; a sixth loss for indicating image semantic differences between a fourth video and a second tag video; and a seventh loss for indicating pixel differences between adjacent video frames in the fourth video.

[0146] According to embodiments of this disclosure, the video encoder shares training parameters for both the video generation task and the resolution optimization task: the target loss function further includes a shared constraint loss for indicating the distribution differences between the third latent code and the fourth latent code in the latent space, wherein the fourth latent code is obtained by compressing the sample first video using a pre-trained video encoder; the video generation task loss further includes a regularization loss for the pre-trained video encoder.

[0147] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0148] It should be noted that the apparatus portion in the embodiments of this disclosure corresponds to the method portion in the embodiments of this disclosure. The description of the apparatus portion is specifically referred to in the method portion, and will not be repeated here.

[0149] Figure 6 A block diagram of an electronic device suitable for implementing a video generation method and apparatus according to embodiments of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0151] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0152] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0153] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0155] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0157] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0158] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0160] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0162] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A video generation method, comprising: Obtain the input text used to indicate the video generation requirements and the input video used as the basis for video generation; The input video is compressed into a first latent code in the latent space using a video encoder; Using the input text as the denoising condition, the first latent code with added random noise is denoised to obtain the second latent code; Using a first video decoder symmetrical to the video encoder, the second latent code is reconstructed into a first video based on the input text; as well as The first video is upsampled using a shared upsampling module to obtain a second video, wherein the resolution of the first video and the input video are the same but smaller than the resolution of the second video, and the shared upsampling module shares the training parameters of the video generation task and the resolution optimization task.

2. The method according to claim 1, wherein, The step of reconstructing the second latent code into the first video based on the input text using a first video decoder symmetrical to the video encoder includes: using the first video decoder... The second latent code is subjected to three-dimensional transposed convolution processing to increase the feature dimension of the second latent code located in the latent space to the feature dimension of the input video, thereby obtaining the first intermediate information; The semantic feature information of the input text and the first intermediate information are fused to obtain fused information; and The fused information is reconstructed into the first video using an activation function.

3. The method according to claim 1, wherein, The step of denoising the first latent code with added random noise using the input text as the denoising condition to obtain the second latent code includes: Using the video generation requirement indicated by the input text as semantic guidance, the first latent code with added random noise is iteratively denoised using a diffusion model to obtain a second latent code that semantically matches the input text.

4. The method according to claim 1 or 2, wherein, The video generation task is used to generate the first video. The resolution optimization task is performed during the training process of the shared upsampling module and is used to reconstruct the third video from the third latent code using the pre-trained second video decoder. The third latent code is obtained by compression based on the first label video. The first label video is a label video with the same resolution as the sample input video. The shared upsampling module is trained using the first sample video and the third sample video, and the first sample video is obtained by reconstructing the second sample latent code.

5. The method according to claim 4, wherein, The pre-trained second video decoder reconstructs the third video from the third latent code in the following manner: The third latent code is subjected to 3D transposed convolution processing to increase the feature dimension of the third latent code located in the latent space to the feature dimension of the first labeled video, thereby obtaining second intermediate information. The pre-trained second video decoder and first video decoder utilize convolutional layers with identical structures to obtain the second intermediate information and the first intermediate information, respectively, and the parameters of the convolutional layers in the pre-trained second video decoder and first video decoder are different. The second intermediate information is reconstructed into the third video using an activation function.

6. The method according to claim 4, wherein, The shared upsampling module is trained in the following manner: The first and third sample videos are upsampled using a pre-trained shared upsampling module to obtain the second and fourth sample videos. A target loss function is determined based on at least two of the sample input video, the sample second video, the fourth video, the first labeled video, the second labeled video, the sample first video, and the third video, wherein the second labeled video has the same resolution as the sample second video, and the first labeled video is obtained by downsampling the second labeled video; as well as Under the constraint of the target loss function, the pre-trained shared upsampling module, the pre-trained video encoder, the pre-trained first video decoder, and the pre-trained second video decoder are jointly fine-tuned until the target loss function converges, thus obtaining the shared upsampling module, the video encoder, the first video decoder, and the second video decoder.

7. The method according to claim 6, wherein, The target loss function includes: video generation task loss and resolution optimization task loss; the video generation task loss includes at least one of the following: a first loss for indicating the pixel reconstruction difference between the sample second video and the second label video; a second loss for indicating the pixel reconstruction difference between the sample first video and the first label video; and a third loss for indicating the image semantic difference between the sample second video and the second label video. The resolution optimization task loss includes at least one of the following: a fourth loss indicating the pixel reconstruction difference between the fourth video and the second labeled video; a fifth loss indicating the pixel reconstruction difference between the third video and the first labeled video; a sixth loss indicating the image semantic difference between the fourth video and the second labeled video; and a seventh loss indicating the pixel difference between adjacent video frames in the fourth video.

8. The method according to claim 6, wherein, The video encoder shares the training parameters for the video generation task and the resolution optimization task: The target loss function further includes a shared constraint loss for indicating the distribution differences between the third and fourth latent codes in the latent space, wherein the fourth latent code is obtained by compressing the sample first video using the pre-trained video encoder; The video generation task loss also includes: regularization loss for the pre-trained video encoder.

9. A video generation apparatus, comprising: The acquisition module is used to acquire the input text indicating the video generation requirements and the input video as the basis for video generation. A compression module is used to compress the input video into a first latent code in the latent space using a video encoder; The denoising module is used to denoise the first latent code with added random noise using the input text as the denoising condition, so as to obtain the second latent code; A reconstruction module is used to reconstruct the second latent code into a first video based on the input text using a first video decoder that is symmetrical to the video encoder. A generation module is used to upsample the first video using a shared upsampling module to obtain a second video, wherein the first video and the input video have the same resolution but the resolution of the second video is smaller, and the shared upsampling module shares the training parameters of the video generation task and the resolution optimization task.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images