High-efficiency video generation method based on memory hierarchical residual error

Abstract

The invention relates to a high-efficiency video generation method based on a memory hierarchical residual error, which comprises the following steps of: setting hyper-parameters including a mean-variance criterion threshold parameter and a de-noising step number; loading a pre-trained video generation model, and selecting a text as a prompt mode; performing hierarchical cache deployment, and setting a threshold value according to the global mean value and the global variance; for a certain step, if the mean value and the variance are both smaller than corresponding threshold values, marking that the step can be subjected to network-level caching; after the network-level cache deployment is completed, performing block-level cache deployment on the de-noising step which is not selected to perform the network-level cache; and generating a video.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and computer vision, and in particular to a method for accelerating and enhancing the quality of a diffusion model for video generation tasks. Background Art

[0002] In the past few years, video generation technology has attracted much attention due to its wide application in various fields. The related generation methods can be mainly divided into several categories, including generation methods based on generative adversarial networks (GANs) [1], generation methods based on autoregression [2], and generation methods based on diffusion models [3]. Among them, generation methods based on diffusion models have shown excellent performance in the field of video generation. This method can iteratively generate videos from Gaussian noise based on given conditions (such as text descriptions). However, methods based on diffusion models still face the problem of high computational cost. These methods require multiple denoising steps, and each denoising step contains a large number of tags that require complex calculations.

[0003] To address these challenges, researchers have proposed various methods to accelerate generation speed, including distillation[4], quantization[5], pruning[6], parallel computing[7], and caching. Among them, cache-based methods are inspired by the memory mechanism of the human brain and attempt to simulate the hierarchical memory organization and information extraction mechanism in the human memory system, aiming to save the intermediate results of the current denoising step and reuse them in adjacent steps. Similar to how the human brain stores and retrieves memories at different levels to improve cognitive efficiency, these cache-based methods can significantly reduce computational costs and do not require training, and therefore have received increasing attention.

[0004] By studying various cache-based methods, we can find that these methods can be viewed as caching residuals at different levels. This hierarchical information processing method is quite similar to the hierarchical memory system of the human brain. (i) Caching layer-level residuals: Similar to the short-term working memory in the human brain, some methods use residuals from each layer in the network for caching. For example, T-GATE[8] explores caching and reusing the predictions of the cross-attention layer between adjacent denoising steps, while PAB[9] selectively performs caching based on the statistical properties of self-attention and cross-attention in different denoising steps. (ii) Caching block-level residuals: Analogous to the episodic memory in the human brain, some methods cache and reuse block-level residuals in several adjacent steps. For example, BlockCaching

[10] selectively reuses the implicit residuals between the input and output of blocks in adjacent steps. Δ-DiT

[11] focuses on the residuals between several consecutive blocks. (iii) Caching network-level residuals: Similar to the long-term semantic memory in the human brain, DeepCache

[12] can be viewed as an approximation using network-level residuals. Specifically, DeepCache caches high-level features of U-Net

[13] and concatenates the cached high-level features with low-level features to reduce computational cost, where the high-level features can be regarded as network-level residuals to some extent.

[0005] These methods primarily focus on using residuals at a single level, failing to fully utilize residuals at different levels, and failing to achieve the collaborative working of multiple levels of memory like the human brain. During information processing, the human brain adaptively coordinates various memory systems based on task requirements, enabling efficient information processing and decision-making. In contrast, existing caching strategies are typically fixed and lack adaptability to data characteristics, resulting in limited acceleration while maintaining generation quality.

[0006] Therefore, a method that hierarchically utilizes multi-level residuals and adaptively selects a caching strategy based on data characteristics is more similar to the human brain's memory organization and information processing mechanisms, and is crucial for further improving video generation efficiency. By simulating the collaborative workings of different memory systems in the human brain, developing more intelligent caching mechanisms will help to simultaneously improve computational efficiency and generation quality.

[0007] References:

[0008] [1] M.Chu, Y.Xie, J.Mayer, L.Leal-Taixé, and N.Thuerey, "Learning temporal coherence via self-supervision for gan-based video generation," ACM TOG, vol.39, no.4, pp.75-1, 2020.

[0009] [2]D.Weissenborn,O. and J.Uszkoreit,“Scaling autoregressivevideo models,”arXiv preprint arXiv:1906.02634,2019.

[0010] [3]J.Ho,T.Salimans,A.Gritsenko,W.Chan,M.Norouzi,and D.J.Fleet,“Videodiffusion models,”in NeurIPS,2022,pp.8633-8646.

[0011] [4]T.Salimans and J.Ho,“Progressive distillation for fast sampling ofdiffusion models,”arXiv preprint arXiv:2202.00512,2022.

[0012] [5]L.Chen,Y.Meng,C.Tang,X.Ma,J.Jiang,X.Wang,Z.Wang,and W.Zhu,“Q-dit:Accurate post-training quantization for diffusion transformers,”arXivpreprint arXiv:2406.17343,2024.

[0013] [6]D.Bolya and J.Hoffman,“Token merging for fast stablediffusion,”inCVPRW,2023,pp.4599-4603.

[0014] [7]M.Li,T.Cai,J.Cao,Q.Zhang,H.Cai,J.Bai,Y.Jia,K.Li,and S.Han,“Distrifusion:Distributed parallel inference for high-resolution diffusionmodels,”in CVPR,2024,pp.7183-7193.

[0015] [8]W.Zhang,H.Liu,J.Xie,F.Faccio,M.Z.Shou,and J.Schmidhuber,“Cross-attention makes inference cumbersome in text-to-image diffusion models,”arXivpreprint arXiv:2404.02747,2024.

[0016] [9]X.Zhao,X.Jin,K.Wang,and Y.You,“Real-time video generation withpyramid attention broadcast,”in ICLR,2025,pp.1–24.

[0017]

[10] F.Wimbauer,B.Wu,E.Schoenfeld,X.Dai,J.Hou,Z.He,A.Sanakoyeu,P.Zhang,S.Tsai,J.Kohler et al.,“Cache me if you can:Accelerating diffusionmodels through block caching,”in CVPR,2024,pp.6211–6220.

[0018]

[11] P.Chen,M.Shen,P.Ye,J.Cao,C.Tu,C.-S.Bouganis,Y.Zhao,and T.Chen,“Δ-DiT:Atraining-free acceleration method tailored for diffusiontransformers,”arXiv preprint arXiv:2406.01125,2024.

[0019]

[12] X.Ma,G.Fang,and X.Wang,“Deepcache:Accelerating diffusion modelsfor free,”in CVPR,2024,pp.15762-15772.

[0020]

[13] O.Ronneberger, P.Fischer, and T.Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical image computing and computer-assisted intervention-MICCAI 2015:18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015, pp.234–241. Summary of the Invention

[0021] Existing cache-based video generation methods usually only utilize residual information at a single level and fail to fully exploit the advantages of residuals at different levels, resulting in a limited balance between acceleration effect and video generation quality. To address the above-mentioned problems of the prior art, the present invention proposes an efficient video generation method based on memory-layered residuals. This method does not require additional training, but systematically utilizes residual information at different levels and adaptively selects a cache strategy based on data characteristics to reduce unnecessary calculations. It can significantly improve the inference speed of the diffusion model while maintaining high-quality video generation. The technical solution is as follows:

[0022] An efficient video generation method based on memory layered residual includes the following steps:

[0023] Step 1: Set hyperparameters, including the mean-variance criterion threshold parameter ɑ and the number of denoising steps S; load a pre-trained video generation model (such as Open-Sora, Open-Sora-Plan, or Latte); and select text as the prompt method;

[0024] Step 2: Tiered Cache Deployment

[0025] a) Perform network-level cache deployment and calculate the network-level residual for each of the S denoising steps of the video generation model. in is the network output, Input for the network;

[0026] b) Generate multiple video samples using different text prompts and calculate the residual difference between adjacent denoising steps to get the mean μ s and variance σ s ;

[0027] c) Calculate the global mean and global variance

[0028] d) Set the threshold θ based on the global mean and global variance μ =α·μ global and θ σ =α·σ global ;

[0029] e) For a certain step in the S denoising steps, if its mean μ s Less than the threshold θ μ And the variance σ s Less than the threshold θ σ , then mark this step as capable of network-level caching;

[0030] Step 3: After completing the network-level cache deployment, perform the following block-level cache deployment for the denoising steps that were not selected for network-level caching:

[0031] a) For each uncached denoising step, calculate the block-level residual in is the block output, Input for the block;

[0032] b) Calculate the mean and variance of different blocks between adjacent steps, and apply the following mean-variance criterion for caching: Calculate the mean square error of all block residuals between adjacent steps to obtain the global mean μ between different blocks at the current step global ' and global variance σ global ', then set the threshold θ μ '=α·μ global ' and θ σ =α·σ global ', by comparing the mean square error of each block residual with the set threshold, determine which blocks are suitable for caching, and perform block-level caching on blocks smaller than the threshold set in this step;

[0033] Step 4: Video generation.

[0034] Furthermore, in step 1, the pre-trained video generation model loaded is Open-Sora, Open-Sora-Plan or Latte.

[0035] Furthermore, α is set to 0.2.

[0036] Furthermore, the method of step 4 is as follows:

[0037] a) Input the text prompt into the text encoder to obtain text features;

[0038] b) Generate random latent space Gaussian noise as initial video features;

[0039] c) Enter the iterative process of S denoising steps:

[0040] For steps marked as performing network-level caching, the network-level residual of the previous step is directly used, skipping the entire network calculation; for steps that do not perform network-level caching but need to perform block-level caching, the block-level residual of the previous step is only used at the corresponding block that needs to perform block-level caching, and other blocks are calculated normally; for steps that neither perform network-level caching nor block-level caching, the complete denoising calculation is performed normally;

[0041] d) After S steps of iteration, the final latent space video features are converted back to pixel space through the decoder to obtain the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Residuals at different levels within the network.

[0043] Figure 2 Flowchart of the present invention DETAILED DESCRIPTION

[0044] The following describes a detailed approach to efficient video generation based on hierarchical residual caching. First, we analyze the different levels of residuals that can be used for caching. Then, we introduce a hierarchical residual caching strategy to accelerate video generation using pre-trained diffusion models.

[0045] In recent years, many cache-based methods have been proposed to accelerate video generation based on diffusion models. We believe that most cache-based methods can be viewed as caching a single level of residuals. In fact, the residuals can be divided into three different levels, as described below:

[0046] 1. Network-level residual: At the network level, the residual is calculated by computing the network output With input The difference between Figure 1 (a) can be expressed as:

[0047]

[0048] For each denoising step of the diffusion model, the output is usually the noise of the prediction, while the input are the latent features of the input video.

[0049] 2. Block-level residual: The network contains multiple different blocks, including spatial blocks and temporal blocks. Figure 1 As shown in (b), the block-level residual can be calculated as:

[0050]

[0051] where r b is the output With input The existing methods Δ-DiT and Block Caching use block-level residuals for caching and reuse.

[0052] 3. Layer-level residual: Layer-level residual represents the difference between the input and output of a specific layer. Figure 1 As shown in (c), for the Transformer model, the layer usually represents a self-attention layer, a cross-attention layer, or an MLP layer. In these layers, there is a clear residual between the input and output, which can be expressed as:

[0053]

[0054] in The input of the representation layer, f l It can be a residual operation, such as an attention operation or an MLP layer. The output is the residual r l and input The existing methods PAB and T-GATE use hierarchical residuals for caching and reuse.

[0055] While caching network-level residuals for approximation can achieve faster speedups, overusing network-level caching can easily lead to degraded generated video quality. Caching residuals from only a few layers or blocks of the model yields limited acceleration, and determining whether to cache a large number of layer-level or block-level residuals in different denoising steps is relatively cumbersome. Therefore, how to leverage these different layers of residuals to accelerate video generation remains an open problem. To address this issue, we propose a hierarchical residual caching method that fully utilizes residuals from different layers.

[0056] Hierarchical residual caching aims to fully utilize residuals at different levels to accelerate video generation without significantly sacrificing quality. As mentioned above, we believe that network-level residuals can significantly accelerate generation. Therefore, we first explore the use of network-level residuals whenever possible without compromising video quality. We then explore the use of block-level residuals to further accelerate generation. We also introduce a mean-variance criterion to determine whether residuals are suitable for caching.

[0057] Mean-variance criterion: The residual cache aims to cache the residual of the current step and reuse it directly in the next few steps. Video generation has S denoising steps, and it is necessary to decide whether a specific denoising step caches the residual for approximate use in adjacent steps. Therefore, it is very important to set the residual cache criterion. A good criterion can effectively accelerate video generation while minimizing quality degradation. Generally speaking, if the residual difference between adjacent denoising steps is small or the same, then the quality will hardly be reduced when the residual of the current diffusion step is used as the residual of the next step. Here, the mean-variance criterion is introduced to find adjacent denoising steps with small residual differences for caching.

[0058] In order to analyze the residual differences between adjacent denoising steps, we use different prompts to generate diverse videos and calculate the residual differences under different prompts. Assume there are M prompts and calculate the residual at denoising step s Afterwards, the mean square error (MSE) between adjacent steps is calculated as The elements represents the MSE between step s and step s+1 for hint m. Then, the mean and variance generated for step s are:

[0059]

[0060] For step s, the mean μ s and variance σ s The smaller it is, the smaller the residual change between the current step and its next step.

[0061] Based on the mean of all steps {μ1,μ2,...,μ s-1} and variance {σ1,σ2,...,σ s-1}, we can get the global mean μ global and the global variance σ global for:

[0062]

[0063] The global mean and variance are used to set the threshold as:

[0064] θ μ =α·μ global (8)

[0065] θ σ =α·σ global (9)

[0066] For the denoising step s, when its mean μ s and variance σ s When both are lower than their respective thresholds, residual caching is performed, which is expressed as:

[0067] μs <θ μ ,σ s <θ σ (10)

[0068] This method directly uses a pre-trained diffusion model to generate video. The proposed memory-based layered residual caching method does not require any parameter learning and therefore does not require training. Next, the proposed memory-based layered residual caching method is applied to a pre-trained video generation model to introduce the specific implementation of the present invention. The specific steps are as follows:

[0069] Step 1: Set relevant hyperparameters including the mean-variance criterion threshold parameter α and the number of denoising steps S; load a pre-trained video generation model such as Open-Sora, Open-Sora-Plan, or Latte; and select text as the prompt method.

[0070] Step 2: Tiered Cache Deployment

[0071] a) First, perform network-level cache deployment. For the S denoising steps of the model, calculate the network-level residual of each step in is the network output, Input for the network.

[0072] b) Generate multiple video samples using different text prompts and calculate the residual difference between adjacent denoising steps to get the mean μ s and variance σ s .

[0073] c) Calculate the global mean and global variance

[0074] d) Set the threshold θ based on the global mean and variance μ =α·μ global and θ σ =α·σ global .

[0075] e) For each denoising step s, if its mean μ s Less than the threshold θ μ And the variance σ s Less than the threshold θ σ , then the step is marked as eligible for network-level caching. It should be noted that caching is not performed on the first few denoising steps because these steps play an important role in generating global structural information and are sensitive to error accumulation.

[0076] Step 3: After completing the network-level cache deployment, perform block-level cache deployment for the denoising steps that are not selected for network-level caching.

[0077] a) For different blocks (such as spatial blocks and temporal blocks) in each uncached denoising step, calculate the block-level residual in is the block output, Input for the block.

[0078] b) Similar to network-level caching, the mean and variance of different blocks between adjacent steps are calculated, and the mean-variance criterion is applied to determine which blocks are suitable for caching.

[0079] Step 4: Video Generation Process

[0080] a) Input the text prompt into the text encoder to obtain text features.

[0081] b) Generate random latent space Gaussian noise as the initial video features.

[0082] c) Enter the iterative process of S denoising steps:

[0083] For steps marked as performing network-level caching, the network-level residuals from the previous step are directly reused, skipping the entire network computation. For steps that do not perform network-level caching but perform block-level caching, only the block-level residuals from the previous step are reused for specific blocks, while computation proceeds normally for other blocks. For steps that perform neither network-level caching nor block-level caching, the full denoising computation proceeds normally. d) After S iterations, the final latent space video features are converted back to pixel space through the decoder to produce the generated video.

[0084] Experimental results show that on Open-Sora, our memory-based hierarchical residual caching method achieves a 2.12x speedup with a VBench score of 78.01%. On Open-Sora-Plan, the method reduces FLOPs from 12032.40T to 3261.53T and latency from 190.06 seconds to 53.69 seconds, achieving a 3.54x speedup. On Latte, the method achieves a 2.09x speedup while maintaining a similar VBench score. In practice, layer-level caching can be implemented after block-level caching. However, experiments show that using layer-level caching after network-level and block-level caching does not show a significant speedup effect, so this experiment focuses on residual caching at the first two levels. Finally, compared with existing acceleration methods, the hierarchical residual caching method achieves the best balance between generation quality and inference speed.

[0085] It should be emphasized that this method can be directly applied to any video generation architecture based on the diffusion model without the need for retraining, while ensuring the quality of video generation and significantly improving the efficiency of video generation.

Claims

1. An efficient video generation method based on memory layered residual, comprising the following steps: Step 1: Set hyperparameters, including the mean-variance criterion threshold parameter α and the number of denoising steps S; load the pre-trained video generation model and select text as the prompt method; Step 2: Tiered Cache Deployment a) Perform network-level cache deployment and calculate the network-level residual for each of the S denoising steps of the video generation model. in is the network output, Input for the network; b) Generate multiple video samples using different text prompts and calculate the residual difference between adjacent denoising steps to get the mean μ s and variance σ s ; c) Calculate the global mean and global variance d) Set the threshold θ based on the global mean and global variance μ =α·μ global and θ σ =α·σ global ; e) For a step in the S denoising steps, if its mean μ s Less than the threshold θ μ And the variance σ s Less than the threshold θ σ , then mark this step as capable of network-level caching; Step 3: After completing the network-level cache deployment, perform the following block-level cache deployment for the denoising steps that were not selected for network-level caching: a) For each uncached denoising step, calculate the block-level residual in is the block output, Input for the block; b) Calculate the mean and variance of different blocks between adjacent steps, and apply the following mean-variance criterion for caching: Calculate the mean square error of all block residuals between adjacent steps to obtain the global mean μ between different blocks at the current step global ' and global variance σ global ', then set the threshold θ μ '=α·μ global ' and θ σ =α·σ global ', by comparing the mean square error of each block residual with the set threshold, determine which blocks are suitable for caching, and perform block-level caching on blocks smaller than the threshold set in this step; Step 4: Video generation.

2. The efficient video generation method based on memory layered residual according to claim 1, characterized in that In step 1, the pre-trained video generation model loaded is Open-Sora, Open-Sora-Plan, or Latte.

3. The efficient video generation method based on memory layered residual according to claim 1, characterized in that Set alpha to 0.

2.

4. The efficient video generation method based on memory layered residual according to claim 1, characterized in that The method for step 4 is as follows: a) Input the text prompt into the text encoder to obtain text features; b) Generate random latent space Gaussian noise as initial video features; c) Enter the iterative process of S denoising steps: For steps marked as performing network-level caching, the network-level residual of the previous step is directly used, skipping the entire network calculation; for steps that do not perform network-level caching but need to perform block-level caching, the block-level residual of the previous step is only used at the corresponding block that needs to perform block-level caching, and other blocks are calculated normally; for steps that neither perform network-level caching nor block-level caching, the complete denoising calculation is performed normally; d) After S steps of iteration, the final latent space video features are converted back to pixel space through the decoder to obtain the generated video.

Images