Video generation method and device, equipment, storage medium and product

Abstract

The invention discloses a video generation method and device, equipment, a storage medium and a product, and relates to the technical field of computer vision. The method comprises the following steps: inputting a video generation cue word into a video generation model; wherein the video generation model comprises a noise generator, a pre-trained diffusion network and a decoder; generating an initial noise video through a noise generator; performing iterative denoising processing on the initial noise video through a diffusion network according to a randomly generated window sequence and a video generation cue word to obtain a target video implicit representation result; each noise video frame in the initial noise video is at least covered by one window in the window sequence; and inputting the target video implicit representation result into a decoder to obtain a target video output by the decoder. The video is randomly divided into a plurality of windows for synchronous denoising through the randomly generated window sequence, so that the accuracy, the stability and the consistency of long video generation are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a video generation method, apparatus, device, storage medium, and product. Background Technology

[0002] Video generation refers to the creation of new, playable video content using algorithms and models. It involves generating new videos featuring characters and consistent styles from input text or images / video clips containing target scenes or characters. The goal of video generation is to enable computers to understand and create dynamic visual content. Existing video generation models can only generate short videos of around 60 frames at a time, which severely limits the generation and application of long videos.

[0003] To overcome this limitation, the main current methods include: (1) segmented video generation methods, such as StreamingT2V. Although this method can maintain the continuity and style consistency of the video, the quality of the generated video segments gradually deteriorates due to the accumulation of video generation errors. At the same time, a new model needs to be trained, which is difficult to be compatible with the existing model ecosystem, limiting its flexibility and scalability in practical applications. (2) Synchronous denoising methods. This method cannot pay attention to global information during denoising. The generated video will have large abrupt changes at the window connection points, which may lead to unstable video quality and affect the clarity and quality of the final video.

[0004] In summary, although existing technologies have made some progress in video generation, problems such as inaccurate and unstable results and poor video consistency still exist in the generation of long videos. Summary of the Invention

[0005] This application provides a video generation method, apparatus, device, storage medium, and product to solve the problems of inaccurate and unstable generation results and poor video consistency in long video generation.

[0006] In a first aspect, embodiments of this application provide a video generation method, including:

[0007] The video generation prompts are input into a pre-trained video generation model; wherein the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder;

[0008] An initial noise video is generated using the noise generator;

[0009] The initial noisy video and the video generation prompt are input into the diffusion network to obtain the diffusion network. The initial noisy video is then iteratively denoised according to the randomly generated window sequence and the video generation prompt to obtain the implicit representation of the target video. Each frame of the initial noisy video is covered by at least one window in the window sequence.

[0010] The implicit representation of the target video is input into the decoder to obtain the target video output by the decoder.

[0011] Secondly, embodiments of this application provide a video generation apparatus, including:

[0012] The prompt input module is used to input video-generated prompts into the video generation model; wherein, the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder;

[0013] A noise video generation module is used to generate an initial noise video through the noise generator;

[0014] The denoising module is used to perform iterative denoising processing on the initial noisy video through the diffusion network, based on a randomly generated window sequence and the video generation prompts, to obtain the implicit representation result of the target video; each frame of the initial noisy video is covered by at least one window in the window sequence;

[0015] The video generation module is used to input the implicit representation result of the target video into the decoder to obtain the target video output by the decoder.

[0016] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the video generation method described in any embodiment of this application.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the video generation method described in any embodiment of this application.

[0021] Fifthly, embodiments of this application provide a computer program product including a computer program that, when executed by a processor, implements the video generation method described in any embodiment of this application.

[0022] The technical solution of this application embodiment involves inputting video generation prompts into a video generation model. The video generation model includes a noise generator, a pre-trained diffusion network, and a decoder. The diffusion network iteratively denoises an initial noisy video based on a randomly generated window sequence and the video generation prompts to obtain an implicit representation of the target video. Each frame of the initial noisy video is covered by at least one window in the window sequence. The implicit representation of the target video is input into the decoder to obtain the target video output by the decoder. By randomly dividing the video into multiple windows using a randomly generated window sequence and synchronously denoising the noisy video frames within each window using the diffusion network, it is ensured that each noisy video frame is covered by at least one window, and the noisy video frames processed by each window can overlap without significant abrupt changes at the window transitions. This improves the accuracy, stability, and consistency of long video generation, solving the problems of inaccurate generation results and poor video consistency in long video generation.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a video generation method provided in Embodiment 1 of this application;

[0026] Figure 2 This is a schematic diagram illustrating the principle of the diffusion network denoising process in a video generation method provided in Embodiment 1 of this application;

[0027] Figure 3 This is a schematic diagram illustrating the effect of a video generation method provided in Embodiment 1 of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a video generation device provided in Embodiment 2 of this application;

[0029] Figure 5A schematic diagram of the structure of an electronic device for implementing the video generation method of this application embodiment. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application. This embodiment is applicable to situations where long videos are generated based on input text. The method can be executed by a video generation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110. Input the video generation prompts into the video generation model; wherein the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder.

[0034] In this context, video generation prompts can be understood as prompts describing the video content to be generated. This content can include: facial expressions, actions, and appearance of characters, as well as the background environment, scene, and style. A video generation model can be understood as a model used to generate the video.

[0035] In this embodiment, the video generation model includes a noise generator, a pre-trained diffusion network, and a decoder. The noise generator is responsible for producing random noise, which is used to generate the initial frames of the video or as the starting point for the generation process. Diffusion networks are a class of generative models based on a probabilistic diffusion process. The core idea is to utilize the inverse process of diffusion to gradually reconstruct a complex data distribution from a simple distribution, improving sample quality by progressively reducing the noise level. During the pre-training process of the diffusion network, Gaussian noise is added to the forward pass to gradually approximate the standard normal distribution of the data. Then, by learning the inverse process, noise is progressively removed, thereby recovering continuous video frames from the noise. The decoder is used to reconstruct video frames from the latent representation. A significant feature of diffusion networks is their ability to generate high-quality samples, especially in image generation tasks, producing realistic details and high-resolution images.

[0036] S120. Generate an initial noise video using a noise generator.

[0037] The initial noisy video can be understood as a video composed of multiple randomly generated frames of noise. Generally, each frame of noise follows a normal distribution, and the length of the initial noisy video can be determined based on the video length or video content input in the video prompt.

[0038] Specifically, the video generation model generates an initial noisy video through a noise generator, which serves as the initial video for progressive denoising and video generation.

[0039] S130. Using a diffusion network, the initial noisy video is iteratively denoised based on a randomly generated window sequence and video generation prompts to obtain the implicit representation of the target video; each frame of the initial noisy video is covered by at least one window in the window sequence.

[0040] The window sequence can be understood as a sequence of multiple windows, with the diffusion network denoising the video frames covered within each window. Each frame of the initial noisy video is covered by at least one window in the window sequence. It should be noted that the multiple windows in the window sequence can be arranged adjacently or overlapped; overlapping noisy video frames may be covered by multiple windows in the window sequence. All windows have equal width, which can be determined based on the upper limit of the number of frames the diffusion network can process.

[0041] The implicit representation of the target video can be understood as the representation of the target video mapped to the latent space, and the target video can be understood as the video that needs to be generated in the end.

[0042] Specifically, the initial noisy video generated by the noise generator and the video generation prompt are input into the diffusion network in the video generation model. Based on the video generation prompt, the diffusion network iteratively denoises the noisy video frames covered by each window in the randomly generated window sequence, and recovers the implicit representation of the target video that matches the video generation prompt from the initial noisy video.

[0043] This embodiment uses a randomly generated window sequence to divide the video into multiple windows. Each window uses a diffusion network for synchronous denoising, ensuring that each video frame is covered by at least one window. Furthermore, the noisy video processed by each window can overlap, and there will be no obvious abrupt changes at the window junctions, thereby ensuring the accuracy and consistency of long video generation.

[0044] S140. Input the implicit representation result of the target video into the decoder to obtain the target video output by the decoder.

[0045] Specifically, the implicit representation of the target video is input into the decoder, and the target video is reconstructed from the implicit representation of the target video in the latent space by the decoder.

[0046] The technical solution of this application embodiment involves inputting video generation prompts into a video generation model. The video generation model includes a noise generator, a pre-trained diffusion network, and a decoder. The noise generator generates an initial noisy video. The diffusion network iteratively denoises the initial noisy video based on a randomly generated window sequence and the video generation prompts to obtain an implicit representation of the target video. Each frame of the initial noisy video is covered by at least one window in the window sequence. The implicit representation of the target video is input into the decoder to obtain the target video output by the decoder. By randomly dividing the video into multiple windows using a randomly generated window sequence and synchronously denoising the noisy video frames within each window using the diffusion network, it is ensured that each noisy video frame is covered by at least one window, and the noisy video frames processed by each window can overlap without significant abrupt changes at the window transitions. This improves the accuracy, stability, and consistency of long video generation.

[0047] As an optional embodiment of this application, S130, the step of iteratively denoising the initial noisy video using the diffusion network based on the randomly generated window sequence and the video generation prompts to obtain the implicit representation result of the target video, includes:

[0048] S131. Using the pre-trained diffusion network, the initial noisy video is denoised according to the randomly generated window sequence and the video generation prompt words to obtain the current video implicit representation result of the current step.

[0049] The current implicit representation result of the video can be understood as the video representation result of the current step in the latent space.

[0050] Specifically, within each window of the randomly generated window sequence, in the current step (e.g., step t), the diffusion network denoises the initial noisy video based on the video generation prompts, obtaining the implicit representation result X of the current video in step t. t ={x t,i}; where i∈[0,N-1], t∈[0,T-1], N represents the total number of frames in the initial noisy video, T represents the total number of denoising steps, and x t,1 Let represent the i-th noisy video frame in step t. The total number of denoising steps T determines the number of iterations the model performs when generating the video. This number can be determined by comprehensively considering factors such as the complexity of the video scene, computational resources, and the performance of the diffusion network. This embodiment of the invention does not impose any restrictions on this.

[0051] S132. Take the current video implicit representation result as the initial noisy video, and return to execute the step of denoising the initial noisy video according to the randomly generated window sequence and the video generation prompt words to obtain the current video implicit representation result of the current step; until the current video implicit representation result of the last step is obtained, determine the current video implicit representation result of the last step as the target video implicit representation result.

[0052] Specifically, the implicit representation result X of the current video at step t is... t As the initial noisy video, return to step S132, which involves denoising the initial noisy video based on the randomly generated window sequence and the video generation prompts to obtain the implicit representation result of the current video in the current step. In step t-1, the initial noisy video X is denoised based on the randomly generated window sequence and the video generation prompts. t Perform denoising processing to obtain the current implicit representation result X of the video at step t-1. t-1 The process continues until the final step (step 0) is obtained, at which point the current implicit video representation result X0 is determined as the target video implicit representation result. After T steps of denoising, a target video implicit representation result with realistic details and high resolution can be obtained.

[0053] It should be noted that the diffusion network in the video generation model of this embodiment only requires basic training and does not need to be trained with a large amount of labeled data. Instead, a training-free method is used in the denoising process. The training-free method is a technique used in generative models, particularly in the field of image and video generation. Unlike traditional generative models, the training-free method does not rely on a pre-trained model but performs computation and optimization in real time during the generation process. The main advantages of this method are its flexibility and real-time performance, enabling the generation of high-quality content without requiring large amounts of data and computational resources. This improves generation speed and flexibility, reduces the cost and complexity of model training, and thus lowers the overall cost of video generation. Furthermore, since no model training is required, it is compatible with the existing model ecosystem, enhancing the flexibility and scalability of video generation models in practical applications.

[0054] In an optional embodiment, step S132, which involves denoising the initial noisy video using the diffusion network based on the randomly generated window sequence and the video generation prompts to obtain the implicit representation result of the current video at the current step, includes:

[0055] A1. Using the diffusion network, the initial noisy video is denoised according to each window in the randomly generated window sequence and the video generation prompt, to obtain the first video frame implicit representation result of the video frame in each window.

[0056] The implicit representation of the first video frame can be understood as the representation of a video frame in the latent space after denoising.

[0057] Specifically, the diffusion network generates prompts based on the video and performs parallel denoising on the noisy video frames covered by the initial noisy video for each window in the randomly generated window sequence, thereby obtaining the implicit representation of the first video frame after denoising within the window.

[0058] For example, if the window sequence contains M windows, and the window width of each window is W, then the initial noisy video can obtain the implicit representation of the first video frame of M×W video frames after each step of denoising.

[0059] A2. For each video frame, the average value of the implicit representation results of the first video frame corresponding to each window is determined as the implicit representation result of the second video frame of the video frame.

[0060] The implicit representation of the second video frame can be understood as the result of averaging the implicit representation of the first video frame within the window.

[0061] Specifically, each video frame is covered by at least one window. The implicit representation of the first video frame after denoising in the window is determined, and the average of at least one implicit representation of the first video frame corresponding to the video frame is calculated to obtain the implicit representation of the second video frame.

[0062] For example, if the window covering the i-th video frame is m1, and the windows covering the (i+3)-th video frame are m1 and m2, then in step t, the implicit representation of the second video frame of the i-th video frame is: The implicit representation of the second video frame of the (i+3)th video frame is: This represents the implicit representation of the first video frame of the i-th video frame in window m1 within the window sequence at step t. This represents the implicit representation of the first video frame of the (i+3)th video frame in window m1 within the window sequence at step t. This represents the implicit representation of the first video frame of the (i+3)th video frame in window m2 of the window sequence at step t.

[0063] A3. Determine the current video implicit representation result based on the second video frame implicit representation result of each video frame.

[0064] Specifically, based on the implicit representation result x of the second video frame from frame 1 to frame N of the initial noisy video at step t. t,i Determine the implicit representation X of the initial noisy video at step t. t .

[0065] In each step of the denoising process of the initial noisy video using a diffusion network, the input initial noisy video is denoised in parallel according to each window in the randomly generated window sequence and the video generation prompt words. The implicit representation results of the denoised video frames are averaged over each frame to obtain the implicit representation result of the current step, which further improves the accuracy and consistency of video generation.

[0066] In a specific example Figure 2 This is a schematic diagram illustrating the principle of the diffusion network denoising process in a video generation method provided in Embodiment 1 of this application. Figure 2 As shown, a diffusion network is used to generate prompts based on a randomly generated window sequence 1 and the video, and then to generate the initial noise sequence X at step T. T After denoising, the implicit representation result X of the current video at step T is obtained. T-1 ; The implicit representation result X of the current video T-1As the initial noise sequence for step T-1, the initial noise sequence X for step T-1 is processed by a diffusion network based on the randomly generated window sequence 2 and the video-generated prompts. T-1 Denoising is performed to obtain the implicit representation of the current video at the (T-1)th digit.

[0067] Window sequence 1 can be represented as:

[0068] [0,W],[S 11 ,S 11 +W],[S 11 +S 12 ,S 11 +S 12 +W],…,[S 11 +S 12 +…,S 11 +S 12 +…+W];

[0069] Window sequence 2 can be represented as:

[0070] [0,W],[S 21 ,S 21 +W],[S 21 +S 22 ,S 21 +S 22 +W],…,[S 21 +S 22 +…,S 21 +S 22 +…+W]; W represents the width of each window in the window sequence, S 11 ,S 12 ,… represents the sliding step size of the window in the randomly generated window sequence 1 at step T; S 21 ,S 22 ,… represents the sliding step size of the window in the window sequence 2 randomly generated in step T.

[0071] This process continues until the current implicit representation result X0 of the video at step 0 is obtained, which is then used as the implicit representation result of the target video.

[0072] In an optional embodiment, A1, the step of performing denoising processing on the initial noisy video according to each window in the randomly generated window sequence and the video generation prompt word through the diffusion network, and obtaining the first video frame implicit representation result of the video frame in each window, includes:

[0073] A11. For each video frame within each window in the window sequence, the self-attention module calculates the self-attention feature of the video frame based on the first query vector of the video frame, the average of the first key vectors of the first key vector of the video frame and the key vector of the anchor frame, and the average of the first value vectors of the first value vector of the video frame and the value vector of the anchor frame; the anchor frame is the first video frame within the window.

[0074] Among them, the first query vector, the first key vector, and the first value vector are respectively the query vector, key vector, and value vector converted from the encoded vector corresponding to the video frame using different weight matrices in the self-attention module.

[0075] Specifically, the first video frame in each window is used as the anchor frame. During the self-attention module's calculation of the self-attention features of each video frame in the window, the anchor frame is injected into the denoising network of that video frame. The value vector and key vector of the video frame, which are jointly determined by the anchor frame and the video frame, are used to replace the value vector and key vector of the video frame in the traditional self-attention module, thus realizing a cross-attention mechanism within the window.

[0076] For example, the formula for calculating the self-attention feature of the i-th video frame within the window is:

[0077]

[0078] Among them, Q1 i K1 i V1 i These are, in order, the first query vector, the first key vector, and the first value vector of the i-th video frame; K0 is the key vector of the anchor frame, and V0 is the value vector of the anchor frame; K i V is the average value of the first bond vector. i Let d be the average value of the first value vector, and d be the dimension of the vector.

[0079] A12. The cross-attention feature is determined by the cross-attention module based on the text embedding vector corresponding to the prompt word generated in the video and the self-attention feature.

[0080] Specifically, the video-generated prompts are encoded using a cross-attention module to obtain a text embedding vector; a second key vector and a second value vector are determined based on the text embedding vector, and the self-attention features of the video frame are determined as the second query vector; cross-attention features are determined based on the second key vector, the second value vector, and the second query vector.

[0081] Among them, the second query vector, the second key vector, and the second value vector are respectively the query vector, key vector, and value vector converted from the encoded vector corresponding to the video frame using different weight matrices in the cross-attention module.

[0082] For example, the formula for calculating the cross-attention feature of the i-th video frame within the window is:

[0083]

[0084] Among them, Q2 i K2 i V2 i The vectors are the second query vector, the second key vector, and the second value vector of the i-th video frame, respectively.

[0085] A13. The cross-attention feature is used as the implicit representation of the first video frame of the video frame.

[0086] This embodiment utilizes a self-attention calculation method based on a cross-attention mechanism to exchange information between different video frames within a window. This ensures that each video frame within the window is aligned with every other video frame within the window, reducing cumulative errors and further improving the coherence and consistency of the video content. Visually, the video appears more realistic and unified, both in terms of dynamic and static features. In videos containing people and backgrounds, it also improves the consistency between the appearance, expressions, and actions of the people in the video and the input prompts, as well as the consistency between the background and the people's clothing.

[0087] In an optional embodiment, A3, the step of determining the current video implicit representation result based on the second video frame implicit representation result of each of the video frames, includes:

[0088] A31. The sequence of implicit representations of the second video frames of each video frame is determined as the implicit representation of the video to be optimized in the current step.

[0089] Among them, the implicit representation result of the video to be optimized can be understood as the implicit representation result of the video that needs to be optimized.

[0090] Specifically, the second video frame implicitly represents the result x of the initial noisy video at step t, from frame 1 to frame N. t,i The constructed sequence is determined as the implicit representation X′ of the current video at step t of the initial noisy video. t , i.e., X′ t ={x t,i}

[0091] A32. Calculate the first loss function value based on the difference between the implicit representation results of the second video frame of every two adjacent video frames; calculate the second loss function value based on the image similarity between each video frame and the anchor frame within each window.

[0092] The first loss function value characterizes the action alignment metric of the implicit representation of video frames. It smooths the motion in the video by aligning the difference between the implicit representations of two consecutive frames in the latent space. The second loss function value characterizes the visual perception inter-frame smoothness metric of the implicit representation of video frames. It reduces the perceptual difference in video content by perceptually aligning each frame of each sliding window with the anchor frame of the current step.

[0093] Specifically, the formula for calculating the first loss function value is as follows:

[0094]

[0095] δx t,i =x t,i -x t,i-1 ;

[0096] δx t,i-1 =x t,i-1 -x t,i-2 ;

[0097] Among them, MotionAlignLoss(X′ t ) represents the implicit representation result X′ of the video to be optimized. t The first loss function value, δx t,i This represents the implicit representation result X′ of the video to be optimized. t The difference between the implicit representation of the second video frame of the adjacent i-th frame and the implicit representation of the second video frame of the (i-1)-th frame, δx t,i X represents the implicit representation result of the video to be optimized. t The difference between the implicit representation of the second video frame of the adjacent (i-1)th frame and the implicit representation of the second video frame of the adjacent (i-2)th frame.

[0098] The formula for calculating the value of the second loss function is:

[0099]

[0100] Among them, LpipsLoss(X′ t ) represents the implicit representation result X′ of the video to be optimized. t The second loss function value, This represents the implicit representation result X′ of the video to be optimized. tThe visually perceived inter-frame smoothness of the implicit representation of the second video frame in the i-th frame within the m-th window and the implicit representation of the second video frame in the 0th frame (i.e., the anchor frame).

[0101] A33. The sum of the first product of the first loss function value and the first hyperparameter, and the second product of the second loss function value and the second hyperparameter is determined as the total loss function value.

[0102] Specifically, the formula for calculating the total loss function value is as follows:

[0103] Loss(X′ t = a1MotionAlignLoss(X′) t )+a2LpipsLoss(X′ t );

[0104] Among them, Loss(X′) t ) represents the total loss function value of the implicit representation result X′ of the video to be optimized, a1 is the first hyperparameter, and a2 is the second hyperparameter.

[0105] A34. Based on the gradient descent method, the implicit representation result of the video to be optimized is optimized according to the total loss function value to obtain the current implicit representation result of the video.

[0106] Specifically, the process of optimizing the implicit representation of the video to be optimized can be represented as follows:

[0107]

[0108] Where, X′ t X represents the implicit representation result of the video to be optimized. t This indicates the implicit representation result of the current video. Let λ represent the gradient operator, and λ represent the learning rate.

[0109] This embodiment optimizes the implicit video representation based on action alignment and visual perception inter-frame smoothness metrics after each denoising step in the diffusion network, reducing abrupt changes between video segments, enhancing video coherence and consistency, and improving video smoothness and naturalness. In generating video scenes containing people, it can separate the generation of facial expressions and head movements, achieving high-quality lip audio alignment.

[0110] As an optional embodiment of this application, the step of randomly generating the window sequence includes: generating an initial window; the starting position of the initial window is the starting frame of the initial noisy video; randomly generating a sliding step sequence, wherein each sliding step in the sliding step sequence is smaller than the window width of the initial window; and sliding the starting position of the initial window toward the ending frame of the initial noisy video in sequence according to the sliding step sequence to obtain a window sequence composed of multiple windows.

[0111] The initial window can be understood as the first window in the window sequence. The starting position of the initial window is the starting frame of the initial noisy video. The width of the initial window can be determined according to the upper limit of the number of frames that the diffusion network can process; however, this embodiment of the invention does not impose any limitation on this. The sliding step sequence can be understood as a sequence composed of multiple sliding steps, where the sliding step is smaller than the width of the initial window.

[0112] Specifically, an initial window is generated based on the preset window width, with the starting position of the initial frame of the initial noisy video.

[0113] To ensure that each frame of the initial noisy video is covered by at least one window in the window sequence, the window sequence needs to completely cover the initial noisy video. Therefore, following the principle that the sliding step size is less than the window width and greater than zero, and the number of frames corresponding to the end position of the last window is greater than the total number of frames in the initial noisy video, a sliding step size sequence is randomly generated. The initial window is then slid from its starting position toward the end frame of the initial noisy video according to the sliding step size sequence, thus obtaining a window sequence consisting of multiple windows.

[0114] For example, if the window width is 16 frames and the total number of frames in the initial noisy video is 90, then the sliding step sequence can be 3, 5, 12, 4, 11, 7, 15, 8, 6, 10. The position corresponding to each window in the window sequence can be [0, 15], [3, 18], [8, 23], [20, 35], [24, 39], [35, 50], [42, 57], [57, 72], [65, 80], [71, 86], [81, 96].

[0115] This embodiment generates a window sequence with a fixed window width and a random sliding step size. This ensures that each noisy video frame is covered by at least one window in the window sequence, while further reducing video conflicts and improving the consistency and coherence of video generation.

[0116] As an optional embodiment of this application, step S120, generating an initial noise video through the noise generator, includes: randomly generating a noise video frame sequence through the noise generator; the length of the noise video frame sequence is equal to the window width of the window in the window sequence, and each noise video frame is normally distributed; copying and splicing the noise video frame sequence to obtain the initial noise video.

[0117] In this context, a noisy video frame sequence can be understood as a sequence composed of multiple noisy video frames. The length of the noisy video frame sequence is equal to the window width of the window in the window sequence, allowing the noisy video frame sequence to fit the window.

[0118] Specifically, the implicit representation of the target video output by the diffusion network is directly related to the initial noisy video, and the noise distribution of the initial noisy video directly affects the quality of the implicit representation of the target video. Therefore, a noisy video frame sequence with a length of window width is randomly generated by a noise generator, and the noisy video frame sequence is copied and spliced ​​to obtain an initial noisy video of length N frames.

[0119] This embodiment can further improve the consistency of style and characters in video generation by arranging similar initial noise in the time dimension.

[0120] Figure 3This is a schematic diagram illustrating the effect of a video generation method provided in Embodiment 1 of this application. In a specific example, the video generation prompt is: "A panda, dressed in a small red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy pawsstrum aminiature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance." The video generation prompt is input into a pre-trained video generation model to generate a video like this... Figure 3 The long video, composed of multiple frames, is visually more realistic and consistent, providing a smoother and more natural viewing experience, thereby enhancing user satisfaction and overall experience.

[0121] Example 2

[0122] Figure 4 This is a schematic diagram of the structure of a video generation device provided in Embodiment 2 of this application. Figure 4 As shown, the device includes: a prompt input module 210, a noisy video generation module 220, a noise reduction module 230, and a video output module 240; wherein,

[0123] The prompt input module 210 is used to input video-generated prompt words into the video generation model; wherein, the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder;

[0124] Noise video generation module 220 is used to generate an initial noise video through the noise generator;

[0125] The denoising module 230 is used to perform iterative denoising processing on the initial noisy video through the diffusion network based on the randomly generated window sequence and the video generation prompt words to obtain the implicit representation result of the target video; each frame of the initial noisy video is covered by at least one window in the window sequence;

[0126] The video output module 240 is used to input the implicit representation result of the target video into the decoder to obtain the target video output by the decoder.

[0127] The technical solution of this application embodiment involves inputting video generation prompts into a video generation model. The video generation model includes a noise generator, a pre-trained diffusion network, and a decoder. An initial noisy video is generated by the noise generator. The initial noisy video and the video generation prompts are input into the diffusion network to obtain an implicit representation of the target video obtained by iterative denoising of the initial noisy video using a randomly generated window sequence and the video generation prompts. Each frame of the initial noisy video is covered by at least one window in the window sequence. The implicit representation of the target video is input into the decoder to obtain the target video output by the decoder. By randomly dividing the video into multiple windows using a randomly generated window sequence, and synchronously denoising the noisy video frames within each window using the diffusion network, it is ensured that each noisy video frame is covered by at least one window, and the noisy video frames processed by each window can overlap without significant abrupt changes at the window transitions. This improves the accuracy, stability, and consistency of long video generation.

[0128] Optionally, the noise reduction module 230 includes:

[0129] The denoising unit is used to perform denoising processing on the initial noisy video through the diffusion network based on the randomly generated window sequence and the video generation prompt words, so as to obtain the current video implicit representation result of the current step;

[0130] The iterative execution unit is used to take the current video implicit representation result as the initial noisy video and return to execute the step of denoising the initial noisy video according to the randomly generated window sequence and the video generation prompt words to obtain the current video implicit representation result of the current step; until the current video implicit representation result of the last step is obtained, the current video implicit representation result of the last step is determined as the target video implicit representation result;

[0131] Optionally, the noise reduction unit includes:

[0132] The noise diffusion subunit is used to perform noise reduction processing on the initial noisy video according to each window in the randomly generated window sequence and the video generation prompt word through the diffusion network, so as to obtain the first video frame implicit representation result of the video frame in each window;

[0133] The video frame implicit representation subunit is used to determine the average value of the first video frame implicit representation result corresponding to each video frame under each window as the second video frame implicit representation result of the video frame;

[0134] The video implicit representation subunit is used to determine the current video implicit representation result based on the second video frame implicit representation result of each of the video frames.

[0135] Optionally, the diffusion network includes a self-attention module and a cross-attention module.

[0136] The noise diffusion subunit is specifically used for:

[0137] The step of performing denoising processing on the initial noisy video according to each window in the randomly generated window sequence and the video generation prompt word through the diffusion network to obtain the first video frame implicit representation result of the video frame in each window includes:

[0138] For each video frame within each window in the window sequence, the self-attention module calculates the self-attention feature of the video frame based on the first query vector of the video frame, the average of the first key vectors of the first key vector of the video frame and the key vector of the anchor frame, and the average of the first value vectors of the first value vector of the video frame and the value vector of the anchor frame; the anchor frame is the first video frame within the window.

[0139] The cross-attention module determines the cross-attention features based on the text embedding vectors corresponding to the prompt words generated in the video and the self-attention features.

[0140] The cross-attention feature is used as the implicit representation of the first video frame of the video frame.

[0141] Optionally, the video implicit representation subunit is specifically used for:

[0142] The sequence of implicit representations of the second video frames of each video frame is determined as the implicit representation of the video to be optimized in the current step.

[0143] The first loss function value is calculated based on the difference between the implicit representation results of the second video frame for every two adjacent video frames.

[0144] The second loss function value is calculated based on the image similarity between each video frame and the anchor frame within each window;

[0145] The sum of the first product of the first loss function value and the first hyperparameter, and the second product of the second loss function value and the second hyperparameter is determined as the total loss function value;

[0146] The implicit representation of the video to be optimized is optimized based on the total loss function value using the gradient descent method to obtain the current implicit representation of the video.

[0147] Optionally, the step of randomly generating the window sequence includes:

[0148] An initial window is generated; the starting position of the initial window is the starting frame of the initial noisy video.

[0149] A random sliding step sequence is generated, wherein each sliding step in the sliding step sequence is smaller than the window width of the initial window;

[0150] According to the sliding step sequence, the starting position of the initial window is slid sequentially towards the ending frame of the initial noisy video to obtain a window sequence consisting of multiple windows.

[0151] Optionally, the noise video generation module 120 is specifically used for:

[0152] The noise generator randomly generates a sequence of noisy video frames; the length of the noisy video frame sequence is equal to the window width of the window in the window sequence, and each noisy video frame follows a normal distribution.

[0153] The noisy video frame sequence is copied and spliced ​​to obtain the initial noisy video.

[0154] The video generation apparatus provided in this application embodiment can execute the video generation method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0155] Example 3

[0156] Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0157] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0158] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0159] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video generation methods.

[0160] In some embodiments, the video generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video generation method by any other suitable means (e.g., by means of firmware).

[0161] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] In some embodiments, the video generation method may be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the video generation method of this application. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0165] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0166] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0167] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0168] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video generation method, characterized in that, include: The video generation prompts are input into a pre-trained video generation model; wherein the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder; An initial noise video is generated using the noise generator; The initial noisy video is iteratively denoised using the diffusion network based on a randomly generated window sequence and the video generation prompts to obtain the implicit representation of the target video; each frame of the initial noisy video is covered by at least one window in the window sequence. The implicit representation of the target video is input into the decoder to obtain the target video output by the decoder.

2. The video generation method according to claim 1, characterized in that, The step of obtaining the implicit representation of the target video by iteratively denoising the initial noisy video through the diffusion network based on the randomly generated window sequence and the video generation prompts includes: The initial noisy video is denoised using the diffusion network based on the randomly generated window sequence and the video generation prompts to obtain the implicit representation of the current video in the current step. The current video implicit representation result is used as the initial noisy video, and the process returns to the step of denoising the initial noisy video according to the randomly generated window sequence and the video generation prompt words to obtain the current video implicit representation result of the current step; until the current video implicit representation result of the last step is obtained, the current video implicit representation result of the last step is determined as the target video implicit representation result; The step of denoising the initial noisy video using the diffusion network based on the randomly generated window sequence and the video generation prompts to obtain the implicit representation result of the current video at the current step includes: The initial noisy video is denoised using the diffusion network according to each window in the randomly generated window sequence and the video generation prompt, to obtain the first video frame implicit representation result of the video frame in each window; For each video frame, the average value of the implicit representation results of the first video frame corresponding to each window is determined as the implicit representation result of the second video frame of the video frame; The current video implicit representation result is determined based on the second video frame implicit representation result of each of the aforementioned video frames.

3. The video generation method according to claim 2, characterized in that, The diffusion network includes a self-attention module and a cross-attention module; the step of using the diffusion network to denoise the initial noisy video according to each window in the randomly generated window sequence and the video generated prompt word to obtain the first video frame implicit representation result of the video frame in each window includes: For each video frame within each window in the window sequence, the self-attention module calculates the self-attention feature of the video frame based on the first query vector of the video frame, the average of the first key vectors of the first key vector of the video frame and the key vector of the anchor frame, and the average of the first value vectors of the first value vector of the video frame and the value vector of the anchor frame; the anchor frame is the first video frame within the window. The cross-attention module determines the cross-attention features based on the text embedding vectors corresponding to the prompt words generated in the video and the self-attention features. The cross-attention feature is used as the implicit representation of the first video frame of the video frame.

4. The video generation method according to claim 3, characterized in that, The step of determining the current video implicit representation result based on the second video frame implicit representation result of each of the video frames includes: The sequence of implicit representations of the second video frames of each video frame is determined as the implicit representation of the video to be optimized in the current step. The first loss function value is calculated based on the difference between the implicit representation results of the second video frame for every two adjacent video frames. The second loss function value is calculated based on the image similarity between each video frame and the anchor frame within each window; The sum of the first product of the first loss function value and the first hyperparameter, and the second product of the second loss function value and the second hyperparameter is determined as the total loss function value; The implicit representation of the video to be optimized is optimized based on the total loss function value using the gradient descent method to obtain the current implicit representation of the video.

5. The video generation method according to any one of claims 1-3, characterized in that, The steps for randomly generating the window sequence include: An initial window is generated; the starting position of the initial window is the starting frame of the initial noisy video. A random sliding step sequence is generated, wherein each sliding step in the sliding step sequence is smaller than the window width of the initial window; According to the sliding step sequence, the starting position of the initial window is slid sequentially towards the ending frame of the initial noisy video to obtain a window sequence consisting of multiple windows.

6. The video generation method according to claim 1, characterized in that, The steps for generating the initial noise video using the noise generator include: The noise generator randomly generates a sequence of noisy video frames; the length of the noisy video frame sequence is equal to the window width of the window in the window sequence, and each noisy video frame follows a normal distribution. The noisy video frame sequence is copied and spliced ​​to obtain the initial noisy video.

7. A video generation apparatus, characterized in that, include: The prompt input module is used to input video-generated prompts into the video generation model; wherein, the video generation model includes: a noise generator, a pre-trained diffusion network, and a decoder; A noise video generation module is used to generate an initial noise video through the noise generator; The denoising module is used to perform iterative denoising processing on the initial noisy video through the diffusion network, based on a randomly generated window sequence and the video generation prompts, to obtain the implicit representation result of the target video; each frame of the initial noisy video is covered by at least one window in the window sequence; The video output module is used to input the implicit representation result of the target video into the decoder to obtain the target video output by the decoder.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video generation method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video generation method according to any one of claims 1-6.

Images