Video generation method, apparatus, device, medium, and product

Abstract

The present disclosure provides a video generation method, device, equipment, medium and product, relating to the technical field of artificial intelligence. The method comprises: inputting a random noise latent code and a target text into a diffusion model to denoise the random noise latent code by taking the target text as semantic guidance, to obtain an initial latent code; performing time dimension expansion and spatial super-resolution reconstruction on the initial latent code respectively to obtain an intermediate latent code; inputting the intermediate latent code and the target text into the diffusion model to determine residual noise, to correct the intermediate latent code based on the residual noise to obtain a target latent code, wherein the residual noise is used to represent the semantic difference between the intermediate latent code and the target text; and mapping the target latent code to a pixel space to generate a target video corresponding to the target text.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a video generation method, apparatus, device, medium, and product. Background Technology

[0002] With the development of generative artificial intelligence, text-to-video generation technology based on diffusion models has gradually become a research hotspot. Related technologies typically employ an end-to-end approach to denoise and reconstruct in high-dimensional pixel space or high-resolution latent code space, using a single-stage network to simultaneously model spatial details and temporal dynamics.

[0003] In the process of realizing the concept of this disclosure, it was found that the related technology has at least the following problems: when generating high-resolution, high-frame-rate videos, it usually faces the problems of high consumption of computing resources, high memory usage, and low inference efficiency. Therefore, it is difficult to balance the generation quality and processing speed under limited hardware conditions, resulting in low generation efficiency. Summary of the Invention

[0004] In view of the above, this disclosure provides a video generation method, apparatus, device, medium, and product.

[0005] One aspect of this disclosure provides a video generation method, comprising: inputting a random noise latent code and target text into a diffusion model to denoise the random noise latent code using the target text as semantic guidance, thereby obtaining an initial latent code; performing temporal dimension expansion and spatial super-resolution reconstruction on the initial latent code to obtain an intermediate latent code; inputting the intermediate latent code and target text into the diffusion model to determine residual noise, thereby correcting the intermediate latent code based on the residual noise to obtain a target latent code, wherein the residual noise is used to characterize the semantic difference between the intermediate latent code and the target text; and mapping the target latent code to a pixel space to generate a target video corresponding to the target text.

[0006] According to embodiments of this disclosure, random noise latent codes and target text are input into a diffusion model to denoise the random noise latent codes using the target text as semantic guidance, thereby obtaining an initial latent code. This includes: using the random noise latent code as the current latent code, iteratively performing the following operations until the number of iterations reaches a first preset number, and outputting an updated initial latent code; inputting the current latent code and target text into the diffusion model and outputting predicted noise, wherein the predicted noise is used to characterize the semantic difference between the current latent code and the target text; updating the current latent code by subtraction based on the predicted noise to obtain the initial latent code, and using the initial latent code as the new current latent code.

[0007] According to embodiments of this disclosure, an intermediate latent code is obtained by performing temporal dimension expansion and spatial super-resolution reconstruction on an initial latent code, including: linear interpolation on the initial latent code to increase the number of frames of the initial latent code to a first target number of frames, obtaining a temporally expanded latent code; spatial super-resolution reconstruction on the temporally expanded latent code through a multi-path super-resolution network to improve the spatial resolution of the temporally expanded latent code to a target resolution, obtaining a spatial super-resolution latent code, wherein the number of frames of the spatial super-resolution latent code is the first target number of frames; and frame interpolation on the spatial super-resolution latent code using an optical flow estimation network to increase the number of frames of the spatial super-resolution latent code to a second target number of frames, obtaining an intermediate latent code.

[0008] According to embodiments of this disclosure, linear interpolation is performed on an initial latent code to increase the number of frames in the initial latent code to a first target number of frames, thereby obtaining a temporally extended latent code. This includes: while maintaining the spatial resolution of the initial latent code, performing trilinear interpolation on adjacent frames in the initial latent code in the temporal dimension to generate multiple intermediate frames; and inserting each intermediate frame between corresponding adjacent frames to increase the number of frames in the initial latent code to a first target number of frames, thereby obtaining a temporally extended latent code.

[0009] According to embodiments of this disclosure, spatial super-resolution reconstruction is performed on a temporally extended latent code using a multi-path super-resolution network to improve the spatial resolution of the temporally extended latent code to a target resolution, thereby obtaining a spatial super-resolution latent code. This includes: inputting the temporally extended latent code into the multi-path super-resolution network to perform pixel recombination of the temporally extended latent code through spatial paths in the multi-path super-resolution network, thereby obtaining a spatial upsampled latent code at the target resolution; while maintaining the inter-frame continuity of the temporally extended latent code, convolving the temporally extended latent code through temporal paths in the multi-path super-resolution network to generate a temporally preserved latent code; and adding the spatial upsampled latent code and the temporally preserved latent code element-wise to obtain the spatial super-resolution latent code.

[0010] According to embodiments of this disclosure, a spatial super-resolution latent code is interpolated using an optical flow estimation network to increase the number of frames in the spatial super-resolution latent code to a second target number of frames, thereby obtaining an intermediate latent code. This includes: inputting the spatial super-resolution latent code into the optical flow estimation network to estimate the optical flow of adjacent frames in the spatial super-resolution latent code, generating forward and backward optical flows; linearly synthesizing at least one intermediate frame between adjacent frames based on the forward and backward optical flows; and inserting the at least one intermediate frame between corresponding adjacent frames to increase the number of frames in the spatial super-resolution latent code from a first target number of frames to a second target number of frames, thereby generating an intermediate latent code.

[0011] According to embodiments of this disclosure, intermediate latent codes and target text are input into a diffusion model to determine residual noise, and the intermediate latent codes are corrected based on the residual noise to obtain target latent codes. This includes: using the intermediate latent codes as reference latent codes, iteratively performing the following operations until the number of iterations reaches a second preset number: inputting the reference latent codes and target text into the diffusion model to output residual noise; superimposing the residual noise onto the reference latent codes to obtain corrected latent codes, and using the corrected latent codes as new reference latent codes; and, when the number of iterations reaches the second preset number, performing pixel recombination on the output corrected latent codes to improve the spatial resolution of the corrected latent codes from the target resolution to the final resolution, thereby obtaining the target latent codes.

[0012] According to embodiments of this disclosure, mapping a target latent code to a pixel space to generate a target video corresponding to the target text includes: dividing the target latent code into multiple overlapping latent code blocks, wherein each latent code block has a preset block size and adjacent latent code blocks have a preset number of overlapping pixels; inputting each latent code block into a video decoder for decoding to obtain corresponding image blocks; merging the overlapping areas between adjacent image blocks according to the preset number of overlapping pixels to generate video frames; and combining the video frames according to a preset time sequence to generate a target video corresponding to the target text.

[0013] Another aspect of this disclosure provides a video generation apparatus, comprising: a latent code denoising module for inputting random noise latent code and target text into a diffusion model to denoise the random noise latent code using the target text as semantic guidance, thereby obtaining an initial latent code; a latent code expansion module for performing temporal dimension expansion and spatial super-resolution reconstruction on the initial latent code to obtain an intermediate latent code; a latent code correction module for inputting the intermediate latent code and target text into the diffusion model to determine residual noise, thereby correcting the intermediate latent code based on the residual noise to obtain a target latent code, wherein the residual noise is used to characterize the semantic difference between the intermediate latent code and the target text; and a video generation module for mapping the target latent code to a pixel space to generate a target video corresponding to the target text.

[0014] Another aspect of this disclosure provides an electronic device comprising:

[0015] One or more processors;

[0016] Memory, used to store one or more programs.

[0017] Specifically, when one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0018] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0019] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0020] According to embodiments of this disclosure, an initial latent code is generated in a low-dimensional latent space using a diffusion model, achieving efficient locking of the video skeleton. Subsequently, temporal expansion and spatial super-resolution operations losslessly expand the low-dimensional latent code into a high-resolution, high-frame-rate intermediate latent code, ensuring motion continuity and spatial consistency. Finally, the diffusion model is reintroduced, using the target text as semantic guidance to perform residual correction on the intermediate latent code, thereby completing the detail completion. This decoupled design—first generating the structure, then deterministically expanding, and finally semantically guiding residual correction—significantly reduces memory usage and computational costs while achieving coordinated optimization of global structure and local details in video generation, improving generation efficiency and visual quality. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 The illustrations depict application scenarios of video generation methods, apparatus, devices, media, and products according to embodiments of the present disclosure.

[0023] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 3 A schematic diagram illustrating the model structure for video generation according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A flowchart illustrating a video generation method according to another embodiment of the present disclosure is shown schematically;

[0026] Figure 5 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 6 A block diagram of an electronic device suitable for implementing a video generation method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0033] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0034] Embodiments of this disclosure provide a video generation method, comprising: inputting random noise latent code and target text into a diffusion model to denoise the random noise latent code using the target text as semantic guidance, thereby obtaining an initial latent code; performing temporal dimension expansion and spatial super-resolution reconstruction on the initial latent code to obtain an intermediate latent code; inputting the intermediate latent code and target text into the diffusion model to determine residual noise, thereby correcting the intermediate latent code based on the residual noise to obtain a target latent code, wherein the residual noise is used to characterize the semantic difference between the intermediate latent code and the target text; and mapping the target latent code to a pixel space to generate a target video corresponding to the target text.

[0035] Figure 1 The illustrations schematically depict application scenarios of video generation methods, apparatuses, devices, media, and products according to embodiments of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the video generation method provided in this embodiment can generally be executed by server 105. Correspondingly, the video generation apparatus provided in this embodiment can generally be located in server 105. The video generation method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the video generation apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the video generation method provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the video generation device provided in this embodiment may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0041] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart illustrating a video generation method according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2 As shown, the method includes operations S210 to S240.

[0044] In operation S210, the random noise latent code and the target text are input into the diffusion model. The target text is used as a semantic guide to denoise the random noise latent code and obtain the initial latent code.

[0045] In operation S220, the initial latent code is subjected to temporal dimension expansion and spatial super-resolution reconstruction to obtain the intermediate latent code.

[0046] In operation S230, the intermediate latent code and the target text are input into the diffusion model to determine the residual noise, and the intermediate latent code is corrected based on the residual noise to obtain the target latent code. The residual noise is used to characterize the semantic difference between the intermediate latent code and the target text.

[0047] In operation S240, the target latent code is mapped to the pixel space to generate the target video corresponding to the target text.

[0048] According to embodiments of this disclosure, a randomly initialized Gaussian noise latent code and target text are input into a diffusion model. The target text can be a text that semantically describes the target video to be generated, such as a video of two dogs playing. After feature extraction of the target text, it is injected into various levels of the diffusion model using a cross-attention mechanism as semantic guidance.

[0049] The diffusion model iteratively denoises the random noise latent code according to a preset scheduler. Specifically, in each iteration, the diffusion model predicts noise and gradually removes it, thereby obtaining an initial latent code that is initially semantically aligned with the target text. The initial latent code maintains a low resolution (e.g., 64×64) in the spatial dimension and corresponds to a fixed frame rate (e.g., 8 frames) in the temporal dimension.

[0050] Subsequently, the initial latent code undergoes temporal expansion and spatial super-resolution reconstruction. Temporal expansion can be achieved by using a temporal upsampling module (such as temporal interpolation or a lightweight video super-resolution network) to increase the number of frames to the target number (e.g., from 8 frames to 16 frames), thereby enhancing temporal coherence. Spatial super-resolution reconstruction utilizes the decoder upsampling block within a spatial super-resolution network or diffusion model to increase the spatial resolution to the target size (e.g., 256×256), thus obtaining an intermediate latent code containing finer-grained spatiotemporal details.

[0051] The intermediate latent code and the target text are then input into the diffusion model again. This time, the diffusion model does not perform complete denoising; instead, it estimates the residual noise between the semantic guidance of the intermediate latent code and the target text through forward inference. The residual noise is the deviation between the noise predicted by the diffusion model and the theoretically noise-free latent code, given the intermediate latent code and the target text, and is used to quantify the semantic difference between the intermediate latent code and the target text.

[0052] Based on residual noise, the intermediate latent code is corrected, for example by using gradient descent or noise subtraction, so that the corrected intermediate latent code is closer to the target text in the feature space, forming the target latent code. The target latent code is mapped to the pixel space frame by frame by the video decoder to reconstruct a continuous video frame sequence, and then post-processed (such as temporal smoothing filtering) to generate a target video that is semantically consistent with the target text and spatiotemporally coherent.

[0053] By utilizing a diffusion model to generate an initial latent code in a low-dimensional latent space, efficient locking of the video skeleton is achieved. Subsequently, temporal expansion and spatial super-resolution operations losslessly expand the low-dimensional latent code into a high-resolution, high-frame-rate intermediate latent code, ensuring motion continuity and spatial consistency. Finally, the diffusion model is reintroduced, using the target text as semantic guidance to perform residual correction on the intermediate latent code, thereby completing the detail completion. Through this decoupled design of first generating the structure, then deterministic expansion, and finally semantically guided residual correction, the system significantly reduces memory usage and computational costs while achieving coordinated optimization of global structure and local details in video generation, improving generation efficiency and visual quality.

[0054] Figure 3 A schematic diagram of the model structure for video generation according to an embodiment of the present disclosure is shown.

[0055] like Figure 3 As shown, the random noise latent code 310 and the target text 320 are input into the diffusion model 330. Under the guidance of text semantics, the random noise latent code 310 is iteratively denoised, and the initial latent code 340 containing the global scene layout and coarse motion trajectory is output.

[0056] Subsequently, the initial latent code 340 enters the temporal spatial expansion module 350. The temporal spatial expansion module 350 sequentially expands the low frame rate latent code to a high frame rate through temporal interpolation, improves the low resolution latent code to a high resolution through a spatial super-resolution network, and further improves the frame rate to the target frame rate through optical flow frame interpolation, thereby generating an intermediate latent code 360 ​​with high spatiotemporal density.

[0057] Next, the intermediate latent code 360 ​​and the target text 320 are input into the diffusion model 330 again. The diffusion model 330 predicts residual noise to quantify the semantic difference between the latent code and the text, and refines the intermediate latent code 360 ​​through multi-step iterative correction to obtain the target latent code 370 that is highly aligned with the text.

[0058] Finally, the target latent code 370 is mapped to the pixel space via the video decoder 380. Through block decoding and overlapping region fusion, continuous video frames are reconstructed to generate the target video 390 corresponding to the target text 320. The entire process uses the low-dimensional latent code space as the core of computation and achieves high computational efficiency while ensuring generation quality through phased, progressive optimization.

[0059] According to embodiments of this disclosure, random noise latent codes and target text are input into a diffusion model to denoise the random noise latent codes using the target text as semantic guidance, thereby obtaining an initial latent code. This includes: using the random noise latent code as the current latent code, iteratively performing the following operations until the number of iterations reaches a first preset number, and outputting an updated initial latent code; inputting the current latent code and target text into the diffusion model and outputting predicted noise, wherein the predicted noise is used to characterize the semantic difference between the current latent code and the target text; updating the current latent code by subtraction based on the predicted noise to obtain the initial latent code, and using the initial latent code as the new current latent code.

[0060] During the iterative denoising process, a first preset number of iterations needs to be set. This first preset number of iterations usually corresponds to the total number of denoising steps preset in the diffusion model sampling scheduler, and the value range is generally 20 to 50 steps. The more steps, the higher the quality of the generated data, but the time consumption will increase accordingly. At the beginning of the iteration, a randomly initialized Gaussian noise latent code is used as the current latent code. Its spatial resolution is usually 1 / 8 or 1 / 16 of the original video frame (such as 64×64), and the time dimension corresponds to the preset initial number of frames (such as 8 frames).

[0061] In each iteration, the current latent code and the target text are input into the diffusion model. This diffusion model typically uses a convolutional architecture and introduces text features as key-value pairs for the cross-attention mechanism in each attention layer. The text features are extracted by a pre-trained text encoder to guide the direction of denoising.

[0062] The diffusion model outputs predicted noise based on the current time step (i.e., the denoising stage of the current iteration). This predicted noise is essentially an estimate of the Gaussian noise remaining in the current latent code by the diffusion model, and also implies the semantic differences between the current latent code and the target text.

[0063] Subsequently, the current latent code is updated by subtraction based on the predicted noise. The specific update method depends on the sampler used. Specifically, the current latent code can be updated by subtracting the product of the predicted noise and the corresponding step size, and the direction can be corrected by combining the noise prediction from the previous step. Alternatively, the update direction of the latent code in the probability flow ordinary differential equation can be calculated based on the noise prediction of the diffusion model.

[0064] The updated latent code is used as the current latent code, and the process proceeds to the next iteration. When the cumulative number of iterations reaches the first preset number, the entire denoising process terminates. At this point, the latent code has gradually evolved from pure noise into an initial latent code with low resolution and low frame rate that is initially aligned with the semantics of the target text, laying a good feature foundation for subsequent temporal extension and spatial super-resolution reconstruction.

[0065] By using iterative subtraction updates to achieve progressive denoising guided by text semantics, the initial latent code is efficiently generated in a low-dimensional latent space. This allows for the rapid locking of the global structure and coarse motion trajectory of the video with low video memory cost, reducing the repeated generation of the overall layout in subsequent processes and laying a solid foundation for efficient detail correction.

[0066] According to embodiments of this disclosure, an intermediate latent code is obtained by performing temporal dimension expansion and spatial super-resolution reconstruction on an initial latent code, including: linear interpolation on the initial latent code to increase the number of frames of the initial latent code to a first target number of frames, obtaining a temporally expanded latent code; spatial super-resolution reconstruction on the temporally expanded latent code through a multi-path super-resolution network to improve the spatial resolution of the temporally expanded latent code to a target resolution, obtaining a spatial super-resolution latent code, wherein the number of frames of the spatial super-resolution latent code is the first target number of frames; and frame interpolation on the spatial super-resolution latent code using an optical flow estimation network to increase the number of frames of the spatial super-resolution latent code to a second target number of frames, obtaining an intermediate latent code.

[0067] Temporal expansion of the initial latent code can be achieved by using linear interpolation to smooth the latent code sequence along the time axis. Intermediate frames are generated by calculating the linearly weighted average of the latent codes in adjacent frames, thereby increasing the number of frames in the initial latent code to the first target frame number, resulting in the temporally expanded latent code. This process operates directly in the latent code space to maintain temporal continuity while reducing the introduction of additional computational overhead.

[0068] Subsequently, the temporally extended latent code is input into a multi-path super-resolution network for spatial super-resolution reconstruction. This multi-path super-resolution network employs a parallel multi-branch structure, including multiple convolutional paths with different dilation rates or kernel sizes to capture multi-scale spatial features. It also incorporates a channel attention mechanism to adaptively weight the outputs of each path, thereby improving the spatial resolution of the temporally extended latent code to the target resolution while maintaining temporal consistency, resulting in a spatial super-resolution latent code. At this point, the number of frames in the spatial super-resolution latent code remains the same as the first target number of frames.

[0069] The spatial super-resolution latent code is processed by frame interpolation using an optical flow estimation network. Specifically, the bidirectional optical flow field between adjacent frames can be calculated by a pre-trained optical flow model, and motion compensation frame interpolation is performed based on the optical flow information. The intermediate frame latent code is synthesized in the temporal dimension, thereby further increasing the number of frames to the second target number of frames, and finally obtaining an intermediate latent code with high frame rate and high spatial resolution.

[0070] A deterministic pipeline comprised of temporal interpolation, spatial super-resolution, and optical flow interpolation efficiently extends spatiotemporal resolution with low computational cost. Linear interpolation rapidly increases the frame rate with zero parameters and zero memory usage; the multipath super-resolution network maintains inter-frame continuity while performing spatial super-resolution; and the optical flow estimation network accurately synthesizes intermediate frames based on motion information. The three technologies work together to ensure smooth motion trajectories, clear spatial details, and stable temporal consistency, providing a high-quality initial foundation for subsequent residual correction.

[0071] According to embodiments of this disclosure, linear interpolation is performed on an initial latent code to increase the number of frames in the initial latent code to a first target number of frames, thereby obtaining a temporally extended latent code. This includes: while maintaining the spatial resolution of the initial latent code, performing trilinear interpolation on adjacent frames in the initial latent code in the temporal dimension to generate multiple intermediate frames; and inserting each intermediate frame between corresponding adjacent frames to increase the number of frames in the initial latent code to a first target number of frames, thereby obtaining a temporally extended latent code.

[0072] The initial latent code is a four-dimensional tensor, typically represented as the number of frames multiplied by the number of channels multiplied by the height multiplied by the width, where the spatial resolution is relatively low. To improve the temporal resolution without destroying the spatial structure, a linear interpolation method is used to upsample along the time axis.

[0073] Specifically, while maintaining the spatial resolution of the initial latent code, trilinear interpolation is performed on two adjacent frames of latent code in the temporal dimension. This involves simultaneous linear weighted calculations in the time, height, and width dimensions to generate an intermediate frame latent code located between the two frames. The interpolation process calculates the number of frames to be inserted based on a preset first target frame number and generates a corresponding number of intermediate frames evenly between each pair of adjacent frames, ensuring a smooth transition of the entire sequence in the temporal dimension. These interpolated intermediate frames are then reassembled with the original frames in chronological order to form the temporally extended latent code after frame number expansion.

[0074] This method utilizes the continuity characteristics in the latent code space and achieves preliminary temporal densification with low computational cost through linear assumptions. At the same time, it reduces semantic incoherence caused by sudden increases in the number of frames, providing a structurally complete and temporally smooth intermediate representation for subsequent more refined spatial super-resolution and optical flow interpolation.

[0075] According to embodiments of this disclosure, spatial super-resolution reconstruction is performed on a temporally extended latent code using a multi-path super-resolution network to improve the spatial resolution of the temporally extended latent code to a target resolution, thereby obtaining a spatial super-resolution latent code. This includes: inputting the temporally extended latent code into the multi-path super-resolution network to perform pixel recombination of the temporally extended latent code through spatial paths in the multi-path super-resolution network, thereby obtaining a spatial upsampled latent code at the target resolution; while maintaining the inter-frame continuity of the temporally extended latent code, convolving the temporally extended latent code through temporal paths in the multi-path super-resolution network to generate a temporally preserved latent code; and adding the spatial upsampled latent code and the temporally preserved latent code element-wise to obtain the spatial super-resolution latent code.

[0076] When performing spatial super-resolution reconstruction of temporal extended latent codes using a multi-path super-resolution network, the temporal extended latent codes are first input into the multi-path super-resolution network, which includes parallel spatial and temporal paths.

[0077] The spatial path typically consists of a series of cascaded deep convolutional layers and pixel recombination layers, enabling efficient upsampling of the input low-resolution latent code. Specifically, the multi-path super-resolution network first expands the number of channels of the temporally extended latent code to a squared multiple of the upsampling factor through convolutional layers. Then, it uses pixel recombination operations to enlarge the spatial dimension to the target resolution while compressing the number of channels back to the original dimension, thereby generating a spatially upsampled latent code. This process can fully utilize the spatial correlation within a single frame to reconstruct sharp high-frequency details.

[0078] Meanwhile, the temporal path is responsible for maintaining the continuity of the temporally extended latent code along the time axis. The temporal path typically uses 3D convolution or temporally separable convolution (i.e., spatial convolution and temporal convolution cascaded) to process the input latent code. The convolutional kernels cover several adjacent frames in the temporal dimension, thereby capturing and enhancing the motion consistency between frames at the feature level. The temporally preserved latent code generated by the temporal path has the same spatial resolution and number of frames as the spatially upsampled latent code.

[0079] Subsequently, the multipath super-resolution network adds the spatial upsampling latent code and the temporal preservation latent code element-wise, achieving complementary fusion of spatial details and temporal information. To further improve the fusion quality, a lightweight refinement network (e.g., composed of several residual blocks) is often added after the addition to jointly optimize the fusion result, ultimately outputting the spatial super-resolution latent code. This spatial super-resolution latent code not only achieves the target resolution but also maintains good inter-frame coherence in the temporal dimension, providing a high-quality foundation for subsequent frame interpolation operations.

[0080] High-fidelity spatial super-resolution is achieved through a dual-path parallel structure, which reduces flicker artifacts caused by traditional frame-by-frame super-resolution and ensures smooth transitions in motion trajectories, thereby improving resolution while maintaining temporal stability and visual consistency of the video.

[0081] According to embodiments of this disclosure, a spatial super-resolution latent code is interpolated using an optical flow estimation network to increase the number of frames in the spatial super-resolution latent code to a second target number of frames, thereby obtaining an intermediate latent code. This includes: inputting the spatial super-resolution latent code into the optical flow estimation network to estimate the optical flow of adjacent frames in the spatial super-resolution latent code, generating forward and backward optical flows; linearly synthesizing at least one intermediate frame between adjacent frames based on the forward and backward optical flows; and inserting the at least one intermediate frame between corresponding adjacent frames to increase the number of frames in the spatial super-resolution latent code from a first target number of frames to a second target number of frames, thereby generating an intermediate latent code.

[0082] Optical flow estimation networks typically employ pre-trained lightweight models to process spatial super-resolution latent codes. During processing, the spatial super-resolution latent code is input into the optical flow estimation network frame by frame. This network uses two adjacent frames as input and performs optical flow calculations along the forward and reverse temporal directions, respectively.

[0083] Specifically, for each pair of adjacent frames, the optical flow estimation network outputs a forward optical flow field from the first frame to the second frame, and a backward optical flow field from the second frame to the first frame. Both optical flow fields are two-dimensional vector graphics of the same size as the frames, depicting the motion trajectory of each pixel between frames.

[0084] Subsequently, the number of intermediate frames to be inserted between adjacent frames is determined based on the second target frame number. For each intermediate moment to be inserted, bidirectional linear synthesis is performed using forward and backward optical flows based on the time ratio of that moment to the preceding and following frames.

[0085] Specifically, the pixels of the previous frame are projected onto the current time step through forward optical flow, while the pixels of the next frame are projected onto the current time step through backward optical flow. The two projection results are then linearly fused according to time weights to generate the complete latent code frame for that intermediate time step. This process is performed in parallel for all adjacent frame pairs to ensure that the intermediate frame retains the spatial details of the original frame while also conforming to the motion continuity between frames.

[0086] After synthesizing all intermediate frames, they are inserted into the original adjacent frames in temporal order, thereby increasing the number of frames of the spatial super-resolution latent code from the first target number of frames to the second target number of frames, and finally generating intermediate latent codes with higher temporal density and smoother motion transitions.

[0087] Optical flow estimation networks are used to estimate the optical flow of adjacent frames and linearly synthesize intermediate frames, achieving efficient and continuous frame interpolation. This ensures smooth and jitter-free video motion after interpolation, providing a high-quality foundation for temporal alignment in subsequent residual correction.

[0088] According to embodiments of this disclosure, intermediate latent codes and target text are input into a diffusion model to determine residual noise, and the intermediate latent codes are corrected based on the residual noise to obtain target latent codes. This includes: using the intermediate latent codes as reference latent codes, iteratively performing the following operations until the number of iterations reaches a second preset number: inputting the reference latent codes and target text into the diffusion model to output residual noise; superimposing the residual noise onto the reference latent codes to obtain corrected latent codes, and using the corrected latent codes as new reference latent codes; and, when the number of iterations reaches the second preset number, performing pixel recombination on the output corrected latent codes to improve the spatial resolution of the corrected latent codes from the target resolution to the final resolution, thereby obtaining the target latent codes.

[0089] During the latent code correction stage, an iterative semantic alignment strategy is adopted, using the intermediate latent code as a reference latent code for further refinement within the latent code space. A second preset number of iterations is set as the number of iterations, which is typically much smaller than the number of steps in the complete denoising process, for example, set to 4 to 8 steps, aiming to fine-tune the latent code with lower computational overhead.

[0090] In each iteration, the current reference latent code and the target text are input into the diffusion model. The diffusion model calculates the noise prediction guided by the text through a cross-attention mechanism, and the output is the residual noise. This residual noise is not the noise residual in the complete denoising process, but rather a noise component predicted by the diffusion model under the current latent code conditions, representing the difference between the current semantics of the latent code and the semantics of the target text. Its intensity and direction indicate how to adjust the latent code to make it closer to the text description.

[0091] Subsequently, the residual noise is superimposed on the reference latent code with a preset coefficient, i.e., a noise subtraction update is performed to obtain the corrected latent code, which is then used as the reference latent code for the next iteration. As the number of iterations increases, the amplitude of the residual noise gradually decreases, and the latent code is gradually pushed closer to the manifold defined by the target text in the semantic space. When the number of iterations reaches the second preset number, the iteration is terminated. At this point, the obtained corrected latent code is semantically highly aligned with the target text, but its spatial resolution remains the target resolution.

[0092] The corrected latent code is input into a pixel reconstruction module, which is typically composed of subpixel convolutional layers or a lightweight super-resolution network. The latent code features are rearranged and upsampled to improve the spatial resolution from the target resolution to the final resolution required by the video, thereby obtaining a target latent code with high spatial clarity and semantic accuracy, laying the foundation for subsequent decoding to generate high-quality video.

[0093] By using a diffusion model to iteratively predict residual noise based on the semantics of the target text and superimposing it onto the intermediate latent code, detail completion can be achieved with only a few iterations. Then, through pixel recombination, the spatial resolution is improved to the final resolution, achieving efficient recovery of high-frequency details while maintaining temporal alignment, and finally generating a clear video that is highly consistent with the semantics of the text.

[0094] Figure 4 A flowchart illustrating a video generation method according to another embodiment of the present disclosure is shown schematically.

[0095] like Figure 4 As shown, the method includes operations S401 to S409.

[0096] In operation S401, the random noise latent code is used as the current latent code and the target text is input into a diffusion model to denoise the current latent code, obtaining the initial latent code. In operation S402, it is determined whether the current denoising iteration count has reached the first preset number; if the result is no, operation S408 is executed; if the result is yes, operation S403 is executed.

[0097] In operation S403, the initial latent code is subjected to temporal dimension expansion and spatial super-resolution reconstruction to obtain an intermediate latent code. In operation S404, the intermediate latent code is used as a reference latent code and input into the diffusion model with the target text to correct the reference latent code, obtaining a corrected latent code. In operation S405, it is determined whether the current correction iteration count has reached the second preset number; if the result is no, operation S409 is executed; if the result is yes, operation S406 is executed.

[0098] In operation S406, the corrected latent code is recombined into pixels to obtain the target latent code. In operation S407, the target latent code is mapped to the pixel space to generate the target video corresponding to the target text. In operation S408, the initial latent code is used as the new current latent code. In operation S409, the corrected latent code is used as the new reference latent code.

[0099] According to embodiments of this disclosure, through a phased decoupling design, coarse-grained denoising is first performed in a low-dimensional latent space with a small number of iterations to generate an initial latent code. Then, a high-resolution intermediate representation is efficiently constructed through deterministic temporal expansion and spatial super-resolution. Finally, semantic details are refined by residual correction. This significantly reduces memory usage and computational cost while achieving coordinated optimization of the global structure and local details of the video, thereby improving generation efficiency and visual quality.

[0100] According to embodiments of this disclosure, mapping a target latent code to a pixel space to generate a target video corresponding to the target text includes: dividing the target latent code into multiple overlapping latent code blocks, wherein each latent code block has a preset block size and adjacent latent code blocks have a preset number of overlapping pixels; inputting each latent code block into a video decoder for decoding to obtain corresponding image blocks; merging the overlapping areas between adjacent image blocks according to the preset number of overlapping pixels to generate video frames; and combining the video frames according to a preset time sequence to generate a target video corresponding to the target text.

[0101] Since the spatial resolution of the target latent code is usually high, directly decoding the entire target latent code may exceed the limitations of video memory or computing resources. Therefore, a strategy of block decoding and fusion is adopted.

[0102] First, the target latent code is spatially divided into multiple overlapping latent code blocks. Each latent code block has a preset block size; for example, a high-resolution latent code can be divided into multiple 256×256 blocks. Furthermore, adjacent blocks have a preset number of overlapping pixels in both the row and column directions, such as a 32-pixel overlap area. This overlapping division method aims to avoid noticeable blocky artifacts during subsequent stitching, ensuring the spatial continuity and smoothness of the video frames.

[0103] Next, each latent code block is input into a video decoder for decoding. This video decoder typically corresponds to the decoder part of a trained video variational autoencoder, capable of mapping the latent code block to its corresponding image block frame by frame. Since the latent code block still retains a complete frame sequence in the time dimension, each image block output after decoding actually corresponds to the content of a set of consecutive video frames in a local spatial region.

[0104] After obtaining all image blocks, the overlapping areas between adjacent image blocks are fused according to a preset number of overlapping pixels. A weighted averaging strategy is used during fusion, for example, weights are linearly assigned according to the distance of the pixel position from the block boundary. Each pixel in the overlapping area is synthesized from the weighted values ​​of corresponding pixels in multiple image blocks. Pixels closer to the block boundary have higher weights, thereby eliminating abrupt changes at the stitching boundary.

[0105] After all overlapping regions are fused, the spatial content of each video frame is formed by stitching together multiple image blocks, creating a complete high-resolution frame. Finally, these fused video frames are combined sequentially according to a preset time order to output a continuous frame sequence. Temporal post-processing, such as mild Gaussian filtering, can be added as needed to further smooth inter-frame differences, ultimately generating a target video that is semantically consistent with the target text, spatially clear, and temporally coherent.

[0106] By dividing the target latent code into multiple overlapping latent code blocks for block decoding and fusing the overlapping areas of adjacent image blocks, the memory usage of a single decoding is effectively reduced, enabling high-resolution video decoding to be executed stably with limited video memory. At the same time, the block boundary effect is eliminated through overlapping fusion, ensuring the visual continuity and overall quality of the output video.

[0107] In one specific embodiment of this disclosure, after receiving the target text input by the user, text encoding is first performed, converting the target text into a semantic feature vector through a text encoder. Simultaneously, the basic size of the latent code space is set according to the target video size. Taking the generation of a high-resolution video of 512×512×16 frames as an example, the initial latent code size is set to 64×64×4 frames, and a Gaussian noise tensor is generated in this latent code space all at once. This noise tensor occupies approximately 128KB of video memory and takes 0.2 milliseconds. All subsequent calculations are performed in this low-dimensional compressed space.

[0108] Subsequently, the draft diffusion stage is entered, employing a lightweight 3D denoising network, namely the diffusion model. The network structure consists of 16 layers, with the number of channels gradually increasing from 320 to 640. Temporal attention mechanisms are introduced only in the last two layers, and denoising is performed in 20 iterations. At each step, the current latent code and text features are input into the network. The network captures coarse-grained motion information through temporal attention, outputs predicted noise, and gradually corrects the latent code. Gradient checkpointing is enabled throughout the process to optimize memory usage, with a peak memory usage of approximately 1.3GB and a total processing time of 2.1 seconds. The final result is an initial latent code that retains the global scene layout and a rough motion trajectory.

[0109] After obtaining the initial latent code, temporal dimension expansion is performed. While keeping the spatial resolution unchanged, trilinear interpolation is performed on adjacent frames in the initial latent code along the time axis to generate intermediate frames and insert them, increasing the number of frames from 4 to 8. This operation takes about 0.3 milliseconds, with no additional increase in video memory, achieving an initial doubling of temporal density.

[0110] Next, spatial super-resolution reconstruction is performed using a learnable multi-path super-resolution network. This network comprises two processing paths: spatial and temporal. The spatial path performs a fourfold upsampling through two pixel rearrangement operations, increasing the spatial resolution from 64×64 to 256×256. The temporal path utilizes 3×3×1 convolutions to preserve continuous features between frames. The outputs of the two paths are added element-wise to obtain the spatial super-resolution latent code. The forward inference computation of this multi-path super-resolution network is approximately 184GB, with a single forward pass taking 35 milliseconds and a peak memory usage of approximately 1.9GB. The spatial resolution directly reaches half the target size.

[0111] After that, a second frame interpolation operation is performed. The frozen optical flow estimation network is used to calculate the optical flow of the 8-frame latent code of size 256×256. Under the guidance of forward and backward optical flow, the 8 intermediate frames are linearly synthesized, further increasing the number of frames from 8 to the target number of 16 frames. This process only performs inference without updating the weights, takes about 4 milliseconds, and increases the video memory by less than 200MB. At this point, all the timing extension is completed and the intermediate latent code is obtained.

[0112] Subsequently, the process enters the residual refinement correction stage, using an intermediate latent code of size 256×256×16 as the reference latent code, and performing correction operations iteratively 5 to 8 times. In each iteration, the reference latent code and the target text are input into the 3D denoising network, and the network outputs residual noise. This noise represents the semantic difference between the current latent code and the target text, and its amplitude is halved compared to the complete noise prediction to accelerate convergence. Then, the residual noise is superimposed on the reference latent code with preset coefficients for updating.

[0113] After completing the iteration, the pixels of the corrected latent code are rearranged to increase the spatial resolution from 256×256 to the final output resolution of 512×512. Each iteration takes about 470 milliseconds, and the total time is about 3.8 seconds. The peak video memory reaches 11.2GB. This stage only supplements high-frequency details and does not change the aligned timing structure to obtain the target latent code.

[0114] Finally, block decoding is performed, spatially dividing the target latent code into multiple overlapping latent code blocks, each measuring 64×64×8 pixels with an 8-pixel overlap between adjacent blocks. Each block is decoded into corresponding image blocks by a video decoder, and then weighted fusion is performed based on the overlapping areas to eliminate splicing artifacts. The video frames are then combined in chronological order. This decoding process takes approximately 1.8 seconds, with peak video memory usage reaching 13.6GB before momentarily dropping back to around 2GB. The final result is a high-definition target video that is semantically consistent with the target text and spatiotemporally coherent. The entire process, from the initial 64×64×4 draft latent code to the final output, takes approximately 7.9 seconds.

[0115] Figure 5 A block diagram of a video generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0116] like Figure 5 As shown, the video generation device 500 includes a latent code denoising module 510, a latent code expansion module 520, a latent code correction module 530, and a video generation module 540.

[0117] The latent code denoising module 510 is used to input random noise latent codes and target text into the diffusion model, so as to use the target text as semantic guidance to denoise the random noise latent codes and obtain the initial latent codes.

[0118] The latent code extension module 520 is used to perform temporal dimension extension and spatial super-resolution reconstruction on the initial latent code to obtain the intermediate latent code.

[0119] The latent code correction module 530 is used to input the intermediate latent code and the target text into the diffusion model, determine the residual noise, and correct the intermediate latent code based on the residual noise to obtain the target latent code. The residual noise is used to characterize the semantic difference between the intermediate latent code and the target text.

[0120] The video generation module 540 is used to map the target latent code to the pixel space and generate a target video corresponding to the target text.

[0121] According to embodiments of this disclosure, the latent code denoising module 510 includes a first iteration module.

[0122] The first iteration module is used to take the random noise latent code as the current latent code and iteratively perform the following operations until the number of iterations reaches a first preset number, and output the updated initial latent code: input the current latent code and the target text into the diffusion model and output the prediction noise, wherein the prediction noise is used to characterize the semantic difference between the current latent code and the target text; based on the prediction noise, the current latent code is updated by subtraction to obtain the initial latent code, and the initial latent code is used as the new current latent code.

[0123] According to embodiments of this disclosure, the latent code extension module 520 includes an interpolation submodule, a supramolecular module, and a frame interpolation submodule.

[0124] The interpolation submodule is used to perform linear interpolation on the initial latent code to increase the number of frames of the initial latent code to the first target number of frames, thereby obtaining the temporally extended latent code.

[0125] The supramolecular module is used to perform spatial super-resolution reconstruction on the temporal extended latent code through a multi-path super-resolution network, so as to improve the spatial resolution of the temporal extended latent code to the target resolution and obtain the spatial super-resolution latent code, wherein the number of frames of the spatial super-resolution latent code is the first target frame number.

[0126] The frame interpolation submodule is used to interpolate the spatial super-resolution latent code using an optical flow estimation network, so as to increase the number of frames of the spatial super-resolution latent code to the number of frames of the second target code, and obtain the intermediate latent code.

[0127] According to embodiments of this disclosure, the interpolation submodule includes a linear interpolation unit and a frame number increment unit.

[0128] The linear interpolation unit is used to perform trilinear interpolation on adjacent frames in the initial latent code in the time dimension while keeping the spatial resolution of the initial latent code unchanged, so as to generate multiple intermediate frames.

[0129] The frame number increment unit is used to insert each intermediate frame between the corresponding adjacent frames to increase the frame number of the initial latent code to the first target frame number, thereby obtaining the temporal extended latent code.

[0130] According to embodiments of this disclosure, the supramolecular module includes a pixel recombination unit, a temporal convolution unit, and an element-addition unit.

[0131] The pixel recombination unit is used to input the temporal extended latent code into the multipath super-resolution network, so as to reconstruct the pixels of the temporal extended latent code through the spatial path in the multipath super-resolution network to obtain the spatial upsampled latent code of the target resolution.

[0132] The temporal convolution unit is used to convolve the temporal extended latent code through the temporal path in the multi-path super-resolution network while maintaining the inter-frame continuity of the temporal extended latent code, so as to generate the temporal-preserving latent code.

[0133] The element-adding unit is used to add the spatial upsampled latent code and the temporally preserved latent code element by element to obtain the spatial super-resolution latent code.

[0134] According to embodiments of this disclosure, the frame interpolation submodule includes an optical flow estimation unit, a frame synthesis unit, and a frame interpolation unit.

[0135] The optical flow estimation unit is used to input the spatial super-resolution latent code into the optical flow estimation network to perform optical flow estimation on adjacent frames in the spatial super-resolution latent code, and generate forward and backward optical flows.

[0136] A frame synthesis unit is used to linearly synthesize at least one intermediate frame between adjacent frames based on the forward optical flow and the backward optical flow.

[0137] The frame insertion unit is used to insert at least one intermediate frame between corresponding adjacent frames to increase the number of frames of the spatial super-resolution latent code from the first target number of frames to the second target number of frames, thereby generating the intermediate latent code.

[0138] According to embodiments of this disclosure, the latent code correction module 530 includes a second iteration module and a pixel recombination module.

[0139] The second iteration module is used to iteratively perform the following operations using the intermediate latent code as the reference latent code, until the number of iterations reaches the second preset number: input the reference latent code and the target text into the diffusion model and output residual noise; superimpose the residual noise onto the reference latent code to obtain the corrected latent code, and use the corrected latent code as the new reference latent code.

[0140] The pixel recombination module is used to reconstruct the pixels of the output modified latent code when the number of iterations reaches a second preset number, so as to improve the spatial resolution of the modified latent code from the target resolution to the final resolution and obtain the target latent code.

[0141] According to embodiments of this disclosure, the video generation module 540 includes a latent code segmentation submodule, a latent code decoding submodule, an image fusion submodule, and a video generation submodule.

[0142] The latent code segmentation submodule is used to divide the target latent code into multiple overlapping latent code blocks, wherein each latent code block has a preset block size and adjacent latent code blocks have a preset number of overlapping pixels.

[0143] The latent code decoding submodule is used to input each latent code block into the video decoder for decoding to obtain the corresponding image block.

[0144] The image fusion submodule is used to fuse the overlapping areas between adjacent image blocks according to a preset number of overlapping pixels to generate video frames.

[0145] The video generation submodule is used to combine video frames according to a preset time sequence to generate a target video corresponding to the target text.

[0146] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0147] For example, any plurality of the latent code denoising module 510, latent code expansion module 520, latent code correction module 530, and video generation module 540 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the latent code denoising module 510, latent code expansion module 520, latent code correction module 530, and video generation module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the latent code denoising module 510, latent code extension module 520, latent code correction module 530, and video generation module 540 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0148] It should be noted that the video generation device part in the embodiments of this disclosure corresponds to the video generation method part in the embodiments of this disclosure. For a detailed description of the video generation device part, please refer to the video generation method part, which will not be repeated here.

[0149] Figure 6 A block diagram of an electronic device suitable for implementing a video generation method according to an embodiment of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0150] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 602 or a program loaded from a storage portion 608 into a random access memory RAM 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0151] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0152] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0153] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0155] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0156] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0157] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the video generation method provided in the embodiments of this disclosure.

[0158] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0160] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0162] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A video generation method, comprising: The random noise latent code and the target text are input into the diffusion model, and the target text is used as semantic guidance to denoise the random noise latent code to obtain the initial latent code. The initial latent code is subjected to temporal dimension expansion and spatial super-resolution reconstruction to obtain the intermediate latent code; The intermediate latent code and the target text are input into the diffusion model to determine residual noise, and the intermediate latent code is corrected based on the residual noise to obtain the target latent code. The residual noise is used to characterize the semantic difference between the intermediate latent code and the target text. The target latent code is mapped to pixel space to generate a target video corresponding to the target text.

2. The method according to claim 1, wherein, The step of inputting random noise latent code and target text into a diffusion model, using the target text as semantic guidance, to denoise the random noise latent code and obtain an initial latent code includes: Using the random noise latent code as the current latent code, iteratively perform the following operations until the number of iterations reaches a first preset number, and then output the updated initial latent code: The current latent code and the target text are input into the diffusion model, and the predicted noise is output, wherein the predicted noise is used to characterize the semantic difference between the current latent code and the target text; Based on the predicted noise, the current latent code is updated by subtraction to obtain the initial latent code, and the initial latent code is used as the new current latent code.

3. The method according to claim 1, wherein, The process of performing temporal dimension expansion and spatial super-resolution reconstruction on the initial latent code to obtain the intermediate latent code includes: Linear interpolation is performed on the initial latent code to increase the number of frames of the initial latent code to the first target number of frames, thereby obtaining the temporally extended latent code; The temporal extended latent code is spatially super-reconstructed using a multi-path super-resolution network to improve the spatial resolution of the temporal extended latent code to the target resolution, thereby obtaining a spatial super-resolution latent code. The number of frames in the spatial super-resolution latent code is the first target number of frames. Using an optical flow estimation network, the spatial super-resolution latent code is frame-interpolated to increase the number of frames of the spatial super-resolution latent code to the second target number of frames, thereby obtaining the intermediate latent code.

4. The method according to claim 3, wherein, The step of performing linear interpolation on the initial latent code to increase the frame number of the initial latent code to the first target frame number, thereby obtaining a time-extended latent code, includes: While keeping the spatial resolution of the initial latent code unchanged, trilinear interpolation is performed on adjacent frames in the initial latent code in the time dimension to generate multiple intermediate frames; Each intermediate frame is inserted between its corresponding adjacent frames to increase the number of frames of the initial latent code to the first target number of frames, thereby obtaining the temporal extended latent code.

5. The method according to claim 3, wherein, The step of performing spatial super-resolution reconstruction on the temporal extended latent code using a multi-path super-resolution network to improve the spatial resolution of the temporal extended latent code to the target resolution, thereby obtaining a spatial super-resolution latent code, includes: The temporal extended latent code is input into the multipath super-resolution network, and the pixels of the temporal extended latent code are recombined through the spatial path in the multipath super-resolution network to obtain the spatial upsampling latent code of the target resolution. While maintaining the inter-frame continuity of the temporal extended latent code, the temporal extended latent code is convolved through the temporal path in the multi-path super-resolution network to generate a temporal-preserving latent code. The spatially upsampled latent code is added element-wise to the temporally preserved latent code to obtain the spatial super-resolution latent code.

6. The method according to claim 3, wherein, The step of using an optical flow estimation network to interpolate frames in the spatial super-resolution latent code to increase the number of frames in the spatial super-resolution latent code to the second target number of frames, thereby obtaining the intermediate latent code, includes: The spatial super-resolution latent code is input into the optical flow estimation network to perform optical flow estimation on adjacent frames in the spatial super-resolution latent code, generating forward optical flow and backward optical flow; Based on the forward optical flow and the backward optical flow, at least one intermediate frame is linearly synthesized between the adjacent frames; The at least one intermediate frame is inserted between corresponding adjacent frames to increase the number of frames of the spatial super-resolution latent code from the first target number of frames to the second target number of frames, thereby generating the intermediate latent code.

7. The method according to claim 1, wherein, The step of inputting the intermediate latent code and the target text into the diffusion model, determining residual noise, and correcting the intermediate latent code based on the residual noise to obtain the target latent code includes: Using the intermediate latent code as a reference latent code, iteratively perform the following operations until the number of iterations reaches a second preset number: The reference latent code and the target text are input into the diffusion model, and residual noise is output. The residual noise is superimposed on the reference latent code to obtain the modified latent code, and the modified latent code is used as the new reference latent code; When the number of iterations reaches the second preset number, the output modified latent code is pixel-recombined to improve the spatial resolution of the modified latent code from the target resolution to the final resolution, thereby obtaining the target latent code.

8. The method according to claim 1, wherein, The step of mapping the target latent code to pixel space to generate a target video corresponding to the target text includes: The target latent code is divided into multiple overlapping latent code blocks, wherein each latent code block has a preset block size and there is a preset number of overlapping pixels between adjacent latent code blocks; Each of the latent code blocks is input into a video decoder for decoding to obtain the corresponding image block; Based on the preset number of overlapping pixels, the overlapping areas between adjacent image blocks are fused to generate video frames; According to a preset time sequence, the video frames are combined to generate a target video corresponding to the target text.

9. A video generation apparatus, comprising: The latent code denoising module is used to input random noise latent codes and target text into a diffusion model, so as to use the target text as semantic guidance to denoise the random noise latent codes and obtain an initial latent code; The latent code extension module is used to perform temporal dimension extension and spatial super-resolution reconstruction on the initial latent code to obtain the intermediate latent code; The latent code correction module is used to input the intermediate latent code and the target text into the diffusion model, determine the residual noise, and correct the intermediate latent code based on the residual noise to obtain the target latent code, wherein the residual noise is used to characterize the semantic difference between the intermediate latent code and the target text; The video generation module is used to map the target latent code to pixel space and generate a target video corresponding to the target text.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images