Video generation method and device based on diffusion model, storage medium and equipment

Abstract

The application discloses a video generation method and device based on a diffusion model, a storage medium and equipment, and belongs to the technical field of deep learning. At each iteration, the current output difference is predicted according to the current time step embedding difference and the segmented fitting function; if the current output difference is less than a predetermined threshold, the current noise embedding is calculated according to the buffered previous noise embedding and the previous output difference; if the current output difference is greater than or equal to the predetermined threshold, the DiT module in the diffusion model is used to process the current time step embedding, the text embedding and the previous noise embedding to obtain the current noise embedding, and the text embedding is obtained by embedding processing the text of the prompt word; and the current noise embedding obtained after the last iteration is decoded to obtain the video. The application can reduce the influence of fitting errors and error accumulation on the video quality through the segmented fitting function, and reuse the previous output difference to calculate the noise embedding when the output difference is small, thereby improving the video generation speed.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a video generation method, apparatus, storage medium and device based on a diffusion model. Background Technology

[0002] When generating videos using cue words, cue words, random noise, and time steps need to be input into the diffusion model. The cue words are converted into text embeddings, random noise into noise embeddings, and time steps into time step embeddings through embedding. The text embeddings, noise embeddings, and time step embeddings are then fed into several DiT (DiffusionTransformer) modules for processing. In each iteration, noise is gradually removed to generate a latent representation of the video. Finally, the latent representation is decoded into the final video by the Wan-Decoder.

[0003] The core bottleneck of the diffusion model stems from the multiple denoising steps in the diffusion process, each of which is time-consuming, and many of these denoising steps actually contribute little to the final result. Research shows that the changes produced at multiple time steps in the denoising process are not uniform, with negligible differences in output at some time steps, thus affecting the video generation speed. Summary of the Invention

[0004] This application provides a video generation method, apparatus, storage medium, and device based on a diffusion model, to address the problem that during the iteration process, the output differences at some time steps are negligible, thus affecting the video generation speed. The technical solution is as follows:

[0005] According to a first aspect of this application, a video generation method based on a diffusion model is provided, the method comprising:

[0006] When using the diffusion model to generate videos based on cue words, the current time step embedding and the previous time step embedding are obtained at each iteration, and the difference between the current time step embedding and the previous time step embedding is calculated.

[0007] The current output difference is predicted based on the embedded difference at the current time step and a preset piecewise fitting function, wherein the piecewise fitting function is a function with different fitting relationships divided according to the time step.

[0008] If the current output difference is less than a predetermined threshold, then the previous noise embedding and the previous output difference are obtained from the cache, and the current noise embedding is calculated based on the previous noise embedding and the previous output difference.

[0009] If the current output difference is greater than or equal to a predetermined threshold, then the text embedding and the previous noise embedding are obtained. The DiT module in the diffusion model is used to process the current time step embedding, the text embedding and the previous noise embedding to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word.

[0010] The video is obtained by decoding the current noise embedding obtained after the last iteration.

[0011] In one possible implementation, the piecewise fitting function is y = f(x, θ)g(t) + 1(1-g(t));

[0012] When t > t res When t = 1;

[0013] When t≤t res When t = 0;

[0014] Where y represents the output difference, x represents the time-step embedded difference, θ represents the fitting parameters, f(x, θ) is the difference fitting function, and t represents the number of time steps. res Preserve the number of steps for the model.

[0015] In one possible implementation, the method further includes:

[0016] Acquire multiple video samples and generate a set of prompt words based on each video sample;

[0017] For each video sample, the text embedding of the prompt word corresponding to the video sample, the noise embedding of random noise, and the time step embedding are input into the DiT module in the diffusion model;

[0018] Calculate the time-step embedding difference and output difference after each iteration;

[0019] The time step embedding differences and the output differences are filtered based on the number of time steps during iteration;

[0020] The time-step embedding differences and output differences of all video samples are fitted with a polynomial to obtain the difference fitting function.

[0021] In one possible implementation, filtering the time-step embedding difference and the output difference based on the time-step number during iteration includes:

[0022] If the number of time steps in this iteration is less than or equal to the number of steps to retain in the model, then the time step embedding difference and the output difference are discarded.

[0023] If the number of time steps in this iteration is greater than the number of steps to retain in the model, then the time step embedding difference and the output difference are retained.

[0024] In one possible implementation, the computation of the time-step embedding difference and output difference generated after each iteration includes:

[0025] After each iteration, obtain the current time step embedding, the previous time step embedding, the current noise embedding, and the previous noise embedding;

[0026] Calculate the time step embedding difference based on the current time step embedding and the previous time step embedding;

[0027] The output difference is calculated based on the current noise embedding and the previous noise embedding.

[0028] In one possible implementation, processing the current time step embedding, the text embedding, and the previous noise embedding using the DiT module in the diffusion model to obtain the current noise embedding includes:

[0029] The DiT module in the diffusion model is used to process the current time step embedding, the text embedding, and the previous noise embedding to obtain the current output difference;

[0030] The current noise embedding is obtained by adding the difference between the previous noise embedding and the current output.

[0031] In one possible implementation, calculating the current noise embedding based on the difference between the previous noise embedding and the previous output includes:

[0032] The current noise embedding is obtained by adding the previous noise embedding and the previous output difference.

[0033] According to a second aspect of this application, a video generation apparatus based on a diffusion model is provided, the apparatus comprising:

[0034] The difference calculation module is used to obtain the current time step embedding and the previous time step embedding in each iteration when generating videos based on prompt words using the diffusion model, and calculate the difference of the current time step embedding based on the current time step embedding and the previous time step embedding.

[0035] The difference prediction module is used to predict the current output difference based on the embedded difference at the current time step and a preset piecewise fitting function, wherein the piecewise fitting function is a function with different fitting relationships divided according to the time step.

[0036] The first generation module is used to obtain the cached previous noise embedding and previous output difference if the current output difference is less than a predetermined threshold, and calculate the current noise embedding based on the previous noise embedding and previous output difference.

[0037] The second generation module is used to obtain the text embedding and the previous noise embedding if the current output difference is greater than or equal to a predetermined threshold. The current time step embedding, the text embedding and the previous noise embedding are processed by the DiT module in the diffusion model to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word.

[0038] The video generation module is used to decode the current noise embedding obtained after the last iteration to obtain the video.

[0039] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the diffusion-based video generation method as described above.

[0040] According to a fourth aspect of this application, a computer device is provided, the computer device including the above-described diffusion-based video generation apparatus.

[0041] The beneficial effects of the technical solution provided in this application include at least the following:

[0042] Since the piecewise fitting function is a function with different fitting relationships divided according to the time step, selecting an appropriate fitting function to predict the current output difference based on the current time step can improve the accuracy of the predicted output difference and effectively reduce the impact of fitting error and error accumulation on video quality. This makes it easier to choose whether to use the buffer to calculate the current noise embedding or use the DiT module to calculate the current noise difference.

[0043] If the current output difference is less than a predetermined threshold, the current noise embedding can be calculated using the cached previous output difference, without having to use the DiT module to calculate the current noise difference. This saves the time consumed by the DiT module in calculating the current noise difference, thereby improving the video generation speed. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1This is a trend chart showing the changes in time-step embedding differences and output differences;

[0046] Figure 2 This is a flowchart of the piecewise fitting function.

[0047] Figure 3 This is a flowchart of a video generation method based on a diffusion model provided in one embodiment of this application;

[0048] Figure 4 This is a flowchart of a video generation method based on a diffusion model provided in one embodiment of this application;

[0049] Figure 5 This is a flowchart illustrating the process of generating a panda video based on panda prompts;

[0050] Figure 6 This is a structural block diagram of a video generation device based on a diffusion model provided in one embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0052] TeaCache employs an innovative caching mechanism. Its core idea is that there is a significant linear relationship between the timestep embedding difference and the output difference. Therefore, by fitting this linear relationship, TeaCache can estimate the output difference between the current timestep and the next timestep in advance.

[0053] Building upon this, TeaCache uses the fitted relationship to estimate the output difference at each time step and sets a predetermined threshold. If the output difference at a certain time step is less than the predetermined threshold, it indicates that the impact of that time step on the final result is small, and the diffusion model does not need to undergo a complete denoising process through the DiT module. Instead, it infers the denoising result for the next time step through residual inference, which can significantly save computation time and accelerate the inference process. Conversely, if the output difference at a certain time step is greater than or equal to the predetermined threshold, the DiT module is used to perform a complete denoising process to ensure the accuracy of the result.

[0054] Traditional uniform caching methods cache the noise embeddings of the diffusion model output at each time step without considering the output differences during the denoising process. TeaCache, on the other hand, selectively caches the noise embeddings of the diffusion model output based on the linear relationship between the time step embedding differences and the output differences at each time step, thereby skipping those time steps with smaller differences and significantly speeding up the inference process.

[0055] While Teacache can improve the utilization of caching strategies through time-step embedding, the following issues still exist:

[0056] (1) Global polynomial fitting has errors: Due to the inconsistent output trends during diffusion, a single global fitting function is difficult to accurately characterize the relationship over the entire time range. For example... Figure 1 As shown, after the fifth time step, the embedding difference and the output difference exhibit a good linear correlation, but are almost unrelated in the first five time steps. Forcing a global polynomial for linear fitting will lead to significant fitting errors, affecting the accuracy of caching decisions.

[0057] (2) Accumulated fitting error affects caching effect: Error caching will skip the output that should be recalculated, causing the initial error to be introduced and continuously accumulate, eventually leading to a larger deviation and significantly reducing the generation quality.

[0058] Therefore, we introduce nonlinear fitting and dynamic buffer control to more accurately characterize the relationship between time step embedding differences and output differences, thereby effectively reducing the impact of fitting errors and error accumulation on generation quality.

[0059] To facilitate understanding, we describe the relationship between the time-step embedding difference and the output difference estimated by TeaCache as a mathematical expression:

[0060] y = f(x, θ) (1)

[0061] Where y represents the output difference, x represents the time-step embedding difference, f() is an expression for y and x (a polynomial in TeaCache), and θ is the fitting parameter in f(), which is fitted across all stages.

[0062]

[0063] Where L() is the loss function, which can be the mean squared error.

[0064] To more accurately fit the expressions for y and x, we introduce a nonlinear component and a dynamic mechanism, which can be represented by the following mathematical expression:

[0065] y = h(x, θ, g(t), t) res (3)

[0066] Where h() is the fitting expression in this application, g() is the nonlinear function, and t is the current denoising step number. res To retain the number of steps for the model, we set it to 5 in the diffusion model (Wan2.1), and θ is the fitting parameter in h(). This parameter can be fitted as follows:

[0067]

[0068] Formula (3) can be expressed in the form of a dynamic caching mechanism, and the resulting piecewise fitting function is as follows:

[0069] y=f(x,θ)g(t)+1(1-g(t)) (5)

[0070] When t > t res When t ≤ t, g(t) = 1; when t ≤ t res When t = 0;

[0071] Where y represents the output difference, x represents the time-step embedded difference, θ represents the fitting parameters, f(x, θ) is the difference fitting function, and t represents the number of time steps. res Preserve the number of steps for the model.

[0072] In the first t with larger errors res In the first step, the DiT module is used to estimate the noise embedding to reduce the impact of early instability. In subsequent steps, a more accurate piecewise fitting function is used to estimate the output difference, thereby determining whether to reuse the cached previous noise embedding and skip the DiT module's calculation. This strategy effectively reduces fitting error during the sampling phase while improving video generation efficiency.

[0073] The following describes the process of generating the phased fitting function.

[0074] (1) Obtain multiple video samples and generate a set of prompt words based on each video sample.

[0075] (2) For each video sample, the text embedding of the prompt word corresponding to the video sample, the noise embedding of random noise, and the time step embedding are input into the DiT module of the diffusion model.

[0076] For each video sample, multiple iterations are performed. In the first iteration, the UMT5 text encoder is used to encode the prompts extracted from the video sample, and the encoded result is then embedded to obtain the text embedding. Random noise is also embedded to obtain the first noise embedding. The Time Encoder is used to encode the first time step, and the encoded result is then embedded to obtain the first time step embedding. The text embedding, the first noise embedding, and the first time step embedding are then processed in the DiT module of the input diffusion model to obtain the first denoising result. In the second iteration, the first denoising result is used as the second noise embedding. The second time step is encoded, and the encoded result is then embedded to obtain the second time step embedding. The text embedding, the second noise embedding, and the second time step embedding are then processed in the DiT module of the input diffusion model to obtain the second denoising result. This process is repeated for each iteration to obtain the final result.

[0077] (3) Calculate the time step embedding difference and output difference generated after each iteration.

[0078] Specifically, calculating the time step embedding difference and output difference generated after each iteration can include: after each iteration, obtaining the current time step embedding, the previous time step embedding, the current noise embedding, and the previous noise embedding; calculating the time step embedding difference based on the current time step embedding and the previous time step embedding; and calculating the output difference based on the current noise embedding and the previous noise embedding.

[0079] Assuming the previous time step embedding is The current time step is embedded as The difference at the current time step

[0080] Assume the previous noise embedding is l t-1 The current noise embedding is l t The current output difference y t =(l t -l t-1 ) / l t .

[0081] (4) Filter the time step embedding difference and output difference based on the number of time steps during iteration.

[0082] Specifically, filtering time-step embedding differences and output differences based on the number of time steps in the iteration can include: if the number of time steps in the current iteration is less than or equal to the number of steps to be retained in the model, then discard the time-step embedding differences and output differences; if the number of time steps in the current iteration is greater than the number of steps to be retained in the model, then retain the time-step embedding differences and output differences.

[0083] If t≤t resThen discard the calculated embedding difference x at the current time step. t Difference from current output y t If t > t res Then the current time step difference x t Difference from current output y t The data is stored as a set in a list.

[0084] (5) Perform polynomial fitting on the time step embedding differences and output differences of all video samples to obtain the difference fitting function.

[0085] Assuming there are 100 video samples, and each video sample is iterated 50 times, then excluding the data from the first 5 iterations, the list `List` can collect a total of 4500 data sets. The computer can then perform a polynomial fitting based on the data in `List` to obtain the difference fitting function f(x, θ).

[0086] like Figure 2 As shown, in simple terms, a total of M sets of prompt words are set. Each set of prompt words is input into the DiT module and iterated N times. The time step difference x is calculated for the output of each DiT module. t The difference between the output and y t If t≤t res If t > t, then pass; res Then List append(x) t y t ), i.e., the time step difference x t The difference between the output and y t The data is stored as a set in a list called List. Finally, the difference fitting function f(x, θ) is obtained by fitting an expression using the data in List.

[0087] like Figure 3 The diagram illustrates a flowchart of a video generation method based on a diffusion model according to an embodiment of this application. This diffusion model-based video generation method can be applied to computer devices. The diffusion model-based video generation method may include:

[0088] Step 301: When generating video based on prompts using the diffusion model, obtain the current time step embedding and the previous time step embedding at each iteration, and calculate the current time step embedding difference based on the current time step embedding and the previous time step embedding.

[0089] In this embodiment, the computer device can use a Time Encoder to encode each time step, and then perform embedding processing on the encoding result to obtain the corresponding time step embedding.

[0090] Assuming the previous time step embedding is The current time step is embedded as The difference at the current time step

[0091] Step 302: Predict the current output difference based on the embedded difference at the current time step and the preset piecewise fitting function. The piecewise fitting function is a function with different fitting relationships divided according to the time step.

[0092] The piecewise fitting function is given by formula (5).

[0093] When predicting the current output difference, the computer device first determines the value of g(t) based on the current time step and the model retain steps, then substitutes the value of g(t) into the piecewise fitting function to obtain the fitting function for the current time step, and finally embeds the difference of the current time step into the fitting function to obtain the current output difference.

[0094] Step 303: If the current output difference is less than a predetermined threshold, obtain the previous noise embedding and the previous output difference from the cache, and calculate the current noise embedding based on the previous noise embedding and the previous output difference.

[0095] If the current output difference is less than the predetermined threshold, it means that the impact of the current output difference is small, and the previous output difference can be directly reused without using the DiT module to calculate the current noise difference. This can save the time consumed by the DiT module to calculate the current noise difference, thereby improving the video generation speed.

[0096] Step 304: If the current output difference is greater than or equal to a predetermined threshold, then obtain the text embedding and the previous noise embedding. Use the DiT module in the diffusion model to process the current time step embedding, text embedding and the previous noise embedding to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word.

[0097] If the current output difference is less than the predetermined threshold, it indicates that the impact of the current output difference is significant, and the DiT module needs to calculate the current noise difference.

[0098] Step 305: Decode the current noise embedding obtained after the last iteration to obtain the video.

[0099] Computer devices can use the Wan-Decoder to decode the current noise embedding obtained after the last iteration, thus generating a video based on the prompt words.

[0100] In summary, the video generation method based on the diffusion model provided in this application improves the accuracy of the predicted output difference by selecting a suitable fitting function based on the current time step to predict the current output difference, since the piecewise fitting function is a function with different fitting relationships divided according to the time step. This effectively reduces the impact of fitting error and error accumulation on video quality, and facilitates the subsequent selection of whether to use the cache to calculate the current noise embedding or to use the DiT module to calculate the current noise difference.

[0101] If the current output difference is less than a predetermined threshold, the current noise embedding can be calculated using the cached previous output difference, without having to use the DiT module to calculate the current noise difference. This saves the time consumed by the DiT module in calculating the current noise difference, thereby improving the video generation speed.

[0102] like Figure 4 The diagram illustrates a flowchart of a diffusion-based video generation method according to an embodiment of this application. This diffusion-based video generation method can be applied to computer devices. The diffusion-based video generation method may include:

[0103] Step 401: When generating video based on prompts using the diffusion model, obtain the current time step embedding and the previous time step embedding at each iteration, and calculate the current time step embedding difference based on the current time step embedding and the previous time step embedding.

[0104] In this embodiment, the computer device can use a Time Encoder to encode each time step, and then perform embedding processing on the encoding result to obtain the corresponding time step embedding.

[0105] Assuming the previous time step embedding is The current time step is embedded as The difference at the current time step

[0106] Step 402: Predict the current output difference based on the embedded difference at the current time step and the preset piecewise fitting function. The piecewise fitting function is a function with different fitting relationships divided according to the time step.

[0107] The piecewise fitting function is given by formula (5).

[0108] When predicting the current output difference, the computer device first determines the value of g(t) based on the current time step and the model retain steps, then substitutes the value of g(t) into the piecewise fitting function to obtain the fitting function for the current time step, and finally embeds the difference of the current time step into the fitting function to obtain the current output difference.

[0109] Step 403: If the current output difference is less than a predetermined threshold, then obtain the previous noise embedding and the previous output difference from the cache, add the previous noise embedding and the previous output difference together to obtain the current noise embedding.

[0110] If the current output difference is less than the predetermined threshold, it means that the impact of the current output difference is small, and the previous output difference can be directly reused without using the DiT module to calculate the current noise difference. This can save the time consumed by the DiT module to calculate the current noise difference, thereby improving the video generation speed.

[0111] Specifically, when generating the current noise difference, the computer device adds the previous noise embedding and the previous output difference to obtain the current noise embedding.

[0112] Assume the previous noise embedding is l t-1 The difference between the previous output and y is t-1 Then the current noise embedding l t =l t-1 +y t-1 .

[0113] Step 404: If the current output difference is greater than or equal to a predetermined threshold, the text embedding and the previous noise embedding are obtained. The DiT module in the diffusion model is used to process the current time step embedding, text embedding and the previous noise embedding to obtain the current output difference. The previous noise embedding and the current output difference are added together to obtain the current noise embedding.

[0114] Text embedding refers to the process of embedding the text of the prompt words.

[0115] If the current output difference is less than the predetermined threshold, it indicates that the impact of the current output difference is significant, and the DiT module needs to calculate the current noise difference.

[0116] Assuming the current time step is embedded as emb time The text is embedded as an emb text The previous noise embedding is l t-1 Then the current noise embedding l t =l t-1 +DiT(l t emb time emb text ).

[0117] If the DiT module is used to calculate the current noise embedding, the computer device also needs to calculate the current output difference y. t =l t -l t-1 The calculation results are cached.

[0118] Step 405: Decode the current noise embedding obtained after the last iteration to obtain the video.

[0119] Computer devices can use the Wan-Decoder to decode the current noise embedding obtained after the last iteration, thus generating a video based on the prompt words.

[0120] In summary, the video generation method based on the diffusion model provided in this application improves the accuracy of the predicted output difference by selecting a suitable fitting function based on the current time step to predict the current output difference, since the piecewise fitting function is a function with different fitting relationships divided according to the time step. This effectively reduces the impact of fitting error and error accumulation on video quality, and facilitates the subsequent selection of whether to use the cache to calculate the current noise embedding or to use the DiT module to calculate the current noise difference.

[0121] If the current output difference is less than a predetermined threshold, the current noise embedding can be calculated using the cached previous output difference, without having to use the DiT module to calculate the current noise difference. This saves the time consumed by the DiT module in calculating the current noise difference, thereby improving the video generation speed.

[0122] In one example, suppose the prompt is "A panda is holding up a cardboard that says 'ucap'", then... Figure 5 As shown, the video generation process is as follows:

[0123] (1) Use the UMT5 encoder to encode the prompt words and embed the encoded results to obtain the text embedding.

[0124] (2) Use the Time Encoder to encode the current time step and embed the encoding result to obtain the current time step embedding.

[0125] (3) Calculate the embedding difference xt of the current time step based on the embedding of the current time step and the embedding of the previous time step, and calculate the embedding difference xt of the current time step. t The difference between the piecewise fitting function y() and the predicted current output y t .

[0126] (4) Determine the current output difference y t If the value is less than the predetermined threshold δ, proceed to step (5); otherwise, proceed to step (6).

[0127] (5) Obtain the previous noise embedding l t-1 Difference from the previous output y t-1 Calculate the current noise embedding l t =lt-1 +yt-1.

[0128] (6) Obtain the current time step embedded emb time Text embedding emb text and the previous noise embedding l t-1 Calculate the current noise embedding l t =l t-1 +DiT(l t emb time emb text ).

[0129] (7) Repeat steps (2)-(6) until the number of iterations reaches N. Use the Wan-Decoder to decode the current noise embedding obtained after the Nth iteration to obtain a video of a panda holding a cardboard sign.

[0130] like Figure 6 The diagram illustrates a structural block diagram of a diffusion-based video generation apparatus according to an embodiment of this application. This diffusion-based video generation apparatus can be applied in a computer device. The apparatus includes:

[0131] The difference calculation module 610 is used to obtain the current time step embedding and the previous time step embedding in each iteration when generating video based on prompt words using the diffusion model, and calculate the difference of the current time step embedding based on the current time step embedding and the previous time step embedding.

[0132] The difference prediction module 620 is used to predict the current output difference based on the embedded difference at the current time step and a preset piecewise fitting function. The piecewise fitting function is a function with different fitting relationships divided according to the time step.

[0133] The first generation module 630 is used to obtain the cached previous noise embedding and the previous output difference if the current output difference is less than a predetermined threshold, and calculate the current noise embedding based on the previous noise embedding and the previous output difference.

[0134] The second generation module 640 is used to obtain the text embedding and the previous noise embedding if the current output difference is greater than or equal to a predetermined threshold. The current time step embedding, text embedding and the previous noise embedding are processed by the DiT module in the diffusion model to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word.

[0135] The video generation module 650 is used to decode the current noise embedding obtained after the last iteration to obtain the video.

[0136] In an optional embodiment, the piecewise fitting function is y = f(x, θ)g(t) + 1(1-g(t));

[0137] When t > t res When t = 1;

[0138] When t≤t res When t = 0;

[0139] Where y represents the output difference, x represents the time-step embedded difference, θ represents the fitting parameters, f(x, θ) is the difference fitting function, and t represents the number of time steps. res Preserve the number of steps for the model.

[0140] In an optional embodiment, the apparatus further includes a function generation module for:

[0141] Acquire multiple video samples and generate a set of prompt words based on each video sample;

[0142] For each video sample, the text embedding of the prompt word corresponding to the video sample, the noise embedding of random noise, and the time step embedding are input into the DiT module of the diffusion model;

[0143] Calculate the time-step embedding difference and output difference after each iteration;

[0144] Filter time step embedding differences and output differences based on the number of time steps during iteration;

[0145] A polynomial fit is performed on the time-step embedding differences and output differences retained for all video samples to obtain the difference fitting function.

[0146] In an optional embodiment, the function generation module is further configured to:

[0147] If the number of time steps in this iteration is less than or equal to the number of steps to retain in the model, then discard the time step embedding differences and output differences.

[0148] If the number of time steps in this iteration is greater than the number of steps to retain in the model, then the time step embedding differences and output differences are retained.

[0149] In an optional embodiment, the function generation module is further configured to:

[0150] After each iteration, obtain the current time step embedding, the previous time step embedding, the current noise embedding, and the previous noise embedding;

[0151] Calculate the time step embedding difference based on the current time step embedding and the previous time step embedding;

[0152] The output difference is calculated based on the current noise embedding and the previous noise embedding.

[0153] In an optional embodiment, the second generation module 640 is further configured to:

[0154] The DiT module in the diffusion model is used to process the current time step embedding, text embedding, and previous noise embedding to obtain the current output difference;

[0155] The difference between the previous noise embedding and the current output is added together to obtain the current noise embedding.

[0156] In an optional embodiment, the first generation module 630 is further configured to:

[0157] The current noise embedding is obtained by adding the difference between the previous noise embedding and the previous output.

[0158] In summary, the video generation apparatus based on the diffusion model provided in this application can improve the accuracy of the predicted output difference by selecting a suitable fitting function based on the current time step to predict the current output difference, since the piecewise fitting function is a function with different fitting relationships divided according to the time step. This effectively reduces the impact of fitting error and error accumulation on video quality, and facilitates the subsequent selection of whether to use the cache to calculate the current noise embedding or to use the DiT module to calculate the current noise difference.

[0159] If the current output difference is less than a predetermined threshold, the current noise embedding can be calculated using the cached previous output difference, without having to use the DiT module to calculate the current noise difference. This saves the time consumed by the DiT module in calculating the current noise difference, thereby improving the video generation speed.

[0160] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the diffusion-based video generation method described above.

[0161] One embodiment of this application provides a computer device, which includes any of the above-described diffusion-based video generation apparatuses.

[0162] It should be noted that the diffusion-based video generation device provided in the above embodiments is only illustrated by the division of the above functional modules when performing diffusion-based video generation. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the diffusion-based video generation device can be divided into different functional modules to complete all or part of the functions described above. In addition, the diffusion-based video generation device and the diffusion-based video generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0163] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0164] The above description is not intended to limit the embodiments of this application. Any adjustments, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A video generation method based on a diffusion model, characterized in that, The method includes: When using the diffusion model to generate videos based on cue words, the current time step embedding and the previous time step embedding are obtained at each iteration, and the difference between the current time step embedding and the previous time step embedding is calculated. The current output difference is predicted based on the embedded difference at the current time step and a preset piecewise fitting function, wherein the piecewise fitting function is a function with different fitting relationships divided according to the time step. If the current output difference is less than a predetermined threshold, then the previous noise embedding and the previous output difference are obtained from the cache, and the current noise embedding is calculated based on the previous noise embedding and the previous output difference. If the current output difference is greater than or equal to a predetermined threshold, then the text embedding and the previous noise embedding are obtained. The DiT module in the diffusion model is used to process the current time step embedding, the text embedding and the previous noise embedding to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word. The video is obtained by decoding the current noise embedding obtained after the last iteration; The piecewise fitting function is y=f(x,θ)g(t)+1(1-g(t)); When t > t res When t = 1; When t≤t res When t = 0; Where y represents the output difference, x represents the time-step embedded difference, θ represents the fitting parameters, f(x, θ) is the difference fitting function, and t represents the number of time steps. res Preserve the number of steps for the model.

2. The video generation method based on the diffusion model according to claim 1, characterized in that, The method further includes: Acquire multiple video samples and generate a set of prompt words based on each video sample; For each video sample, the text embedding of the prompt word corresponding to the video sample, the noise embedding of random noise, and the time step embedding are input into the DiT module in the diffusion model; Calculate the time-step embedding difference and output difference after each iteration; The time step embedding differences and the output differences are filtered based on the number of time steps during iteration; The time-step embedding differences and output differences of all video samples are fitted with a polynomial to obtain the difference fitting function.

3. The video generation method based on the diffusion model according to claim 2, characterized in that, The step of filtering the time step embedding difference and the output difference based on the time step number during iteration includes: If the number of time steps in this iteration is less than or equal to the number of steps to retain in the model, then the time step embedding difference and the output difference are discarded. If the number of time steps in this iteration is greater than the number of steps to retain in the model, then the time step embedding difference and the output difference are retained.

4. The video generation method based on the diffusion model according to claim 2, characterized in that, The calculation of the time-step embedding difference and output difference generated after each iteration includes: After each iteration, obtain the current time step embedding, the previous time step embedding, the current noise embedding, and the previous noise embedding; Calculate the time step embedding difference based on the current time step embedding and the previous time step embedding; The output difference is calculated based on the current noise embedding and the previous noise embedding.

5. The video generation method based on a diffusion model according to claim 1, characterized in that, The process of using the DiT module in the diffusion model to process the current time step embedding, the text embedding, and the previous noise embedding to obtain the current noise embedding includes: The DiT module in the diffusion model is used to process the current time step embedding, the text embedding, and the previous noise embedding to obtain the current output difference; The current noise embedding is obtained by adding the difference between the previous noise embedding and the current output.

6. The video generation method based on the diffusion model according to claim 1, characterized in that, The step of calculating the current noise embedding based on the difference between the previous noise embedding and the previous output includes: The current noise embedding is obtained by adding the previous noise embedding and the previous output difference.

7. A video generation device based on a diffusion model, characterized in that, The device includes: The difference calculation module is used to obtain the current time step embedding and the previous time step embedding in each iteration when generating videos based on prompt words using the diffusion model, and calculate the difference of the current time step embedding based on the current time step embedding and the previous time step embedding. The difference prediction module is used to predict the current output difference based on the embedded difference at the current time step and a preset piecewise fitting function, wherein the piecewise fitting function is a function with different fitting relationships divided according to the time step. The first generation module is used to obtain the cached previous noise embedding and previous output difference if the current output difference is less than a predetermined threshold, and calculate the current noise embedding based on the previous noise embedding and previous output difference. The second generation module is used to obtain the text embedding and the previous noise embedding if the current output difference is greater than or equal to a predetermined threshold. The current time step embedding, the text embedding and the previous noise embedding are processed by the DiT module in the diffusion model to obtain the current noise embedding. The text embedding is obtained by embedding the text of the prompt word. The video generation module is used to decode the current noise embedding obtained after the last iteration to obtain the video; The piecewise fitting function is y=f(x,θ)g(t)+1(1-g(t)); When t > t res When t = 1; When t≤t res When t = 0; Where y represents the output difference, x represents the time-step embedded difference, θ represents the fitting parameters, f(x, θ) is the difference fitting function, and t represents the number of time steps. res Preserve the number of steps for the model.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the video generation method based on the diffusion model as described in any one of claims 1 to 6.

9. A computer device, characterized in that, The computer device includes: the video generation apparatus based on the diffusion model as described in claim 7.

Images