Video generation model inference optimization method based on activation tiling and sparse computation cooperation

Abstract

This invention discloses a video generation model inference optimization method based on activation tiling and sparse computation, belonging to the field of video generation model inference optimization technology. The method includes the following steps: 1) Inference process analysis: Analyzing the memory overhead and computation time ratio of each operator in the model during the inference process, using the roofline model to identify whether the operator is limited by memory transfer speed or computation hardware speed, thereby obtaining the memory overhead bottleneck operator and the computation time bottleneck operator in the inference process; 2) Activation tiling: For the memory overhead bottleneck operator, using an activation tiling method that combines feature slicing and multi-operator fusion to reduce the memory overhead of activation value storage and temporary buffers; 3) Adaptive sparse computation: Adaptively selecting whether to use a sparse mask based on the characteristics of the model layer attention map, reducing the computational load of model inference while maximizing inference accuracy.

Description

Technical Field

[0001] This invention belongs to the field of video generation model inference optimization technology, specifically involving a video generation model inference optimization method based on the collaboration of activation tiling and sparse computation. Background Technology

[0002] Advances in video generation have profoundly changed people's daily lives. With the help of Video Diffusion Transformers (VDiTs), video generation now automatically transforms text descriptions into high-quality videos, incorporating hyper-realistic motion, lighting, and engaging narratives. However, this generation process involves significant computation, resulting in long inference processing times and substantial hardware memory consumption. The complex computational patterns in video generation models, such as the Spatio-temporal 3D Full Attention mechanism, are the main reason for the high computational and memory requirements of model inference. Optimizing algorithms to accelerate inference and reduce memory overhead is a cost-effective solution, especially suitable for users with a single consumer-grade GPU.

[0003] However, balancing computation time reduction, peak memory consumption mitigation, and maintaining video generation quality is challenging. Existing methods often over-optimize one metric at the expense of others: 1) Inference acceleration schemes that prioritize reducing computation time, such as activation caching and sparse computation methods, often increase memory consumption while improving computation speed; 2) Weighted CPU offloading methods that maintain quality and reduce memory overhead can lead to a significant increase in computation time; 3) Model compression schemes that simultaneously reduce computation time and memory overhead, such as quantization, model distillation, and weight pruning methods, often result in a significant drop in video generation quality, requiring substantial time and data for fine-tuning to restore quality.

[0004] To address the aforementioned issues and reduce memory overhead and computation time while ensuring video generation quality, this invention proposes a video generation model inference optimization method based on the collaboration of activation tiling and sparse computation. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the present invention aims to provide a video generation model inference optimization method based on the collaboration of activation tiling and sparse computation. By analyzing the characteristics of the inference process of the video generation model, activation tiling and sparse computation are applied to the memory overhead bottleneck and computation time bottleneck of the inference process, respectively, thereby significantly reducing the memory overhead and computation time of the model inference while maintaining the quality of video generation.

[0006] To achieve the above objectives, the present invention can be implemented using the following specific technical solutions: The aforementioned video generation model inference optimization method based on the collaboration of activation tiling and sparse computation includes the following steps: 1) Inference process analysis: Using the PyTorch Profiler library and PyTorch Snapshot function, we analyze the memory overhead and computation time ratio of each operator in the model during the inference process. We use the Roofline Model in NVIDIA Nsight Compute to identify whether the operator is limited by memory transfer speed (Memory-bound) or computation hardware speed (Computation-bound). This allows us to obtain the memory overhead bottleneck operator (QK normalization operator, rotation position encoding operator, and three operators of the feedforward module in VDiTs) and the computation time bottleneck operator (QKV Attention operator in VDiTs).

[0007] 2) Activation Tile: For operators with bottleneck memory overhead, an activation tile method combining feature slicing and multi-operator fusion is adopted to reduce the memory overhead of activation value storage and temporary buffers. The activation tile method includes: feature slicing decomposes the input features of the QK normalization operator, rotation position encoding operator, and feedforward module into smaller sub-features; after feature slicing, the QK normalization operator and rotation position encoding operator are fused into one operator group, and the three operators of the feedforward module are fused into another operator group. Multi-operator fusion avoids reassembling all sub-feature calculation results into full-size activation values ​​after a single operator calculation is completed. Instead, it directly passes the sub-feature calculation results to adjacent operators to avoid large intermediate results, thereby reducing the memory overhead of temporary buffers.

[0008] 3) Adaptive Sparse Computation: Based on the characteristics of the attention maps in the model layers, the use of sparse masks is adaptively selected to reduce the computational cost of model inference while maximizing inference accuracy. Specifically, this involves calculating the attention maps (i.e., the sparse maps of different layers of the video diffusion transformer)... Analyzing the values ​​of attention maps reveals that not all attention maps possess a sparse shape. Existing sparse computation methods force the addition of sparse masks to all attention maps, causing attention maps without sparse shapes to discard many important values, resulting in significant inference accuracy loss. This invention proposes adaptive sparsity computation, adding sparse masks only to attention maps with sparse shapes, while leaving the remaining attention maps unmasked.

[0009] Furthermore, in the inference process analysis, operators whose activation value storage overhead is higher than the average and limited by the video memory transfer speed are identified as inference video memory bottlenecks, and operators that occupy the largest proportion of inference time and are limited by the computing hardware speed are identified as computing time bottlenecks.

[0010] Furthermore, in the activation tiling, feature slices are performed based on the operator-calculated features, and adjacent operators with consistent calculated features are merged into an operator group; the QK normalization operator and the rotation position encoding operator maintain the integrity of the last two dimensions and perform segmentation in the dimensions other than the last two dimensions; the input projection operator, Gaussian error linear unit operator, and output projection operator of the feedforward module maintain the last dimension of the feature and perform segmentation in the dimensions other than the last dimension.

[0011] Furthermore, in adaptive sparse computation, layers suitable for sparse masks are identified by comparing the P-value distributions of two sampling regions in the attention map. Sparse masks are added only to layers with sparse shapes, thereby minimizing video generation quality loss and reducing computational cost. The core criterion for identification is: if the sum of the P-values ​​of region A in the sparse mask is greater than or equal to the sum of the P-values ​​of region A in the sparse mask, then... A The sum of P values ​​for regions B not significantly exceeding those in the sparse mask is not (Sum(P)). B If the attention map of that layer is not suitable for using sparse masks, then the attention map of that layer is considered unsuitable for use.

[0012] Compared with the prior art, the present invention has the following advantages: This invention utilizes a video generation model inference optimization method to achieve a better balance between generation quality, memory overhead, and computation time, thereby enabling the deployment of high-quality video generation models and fast inference on consumer-grade GPUs. Attached Figure Description

[0013] Figure 1 This is a diagram showing the analysis results of the reasoning process of this invention; Figure 2 This is a schematic diagram of the activation tiling method of the present invention; Figure 3 This is a comparison diagram of different attention maps in this invention; Figure 4 This is the attention graph sparse pattern preference statistical graph of the present invention; Figure 5 This is a comparison chart of the inference overhead index of this invention with other technical indicators. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0015] This invention provides an inference optimization method for video generation models based on the collaboration of activation tiling and sparse computation. It analyzes and identifies memory overhead bottleneck operators and computation time bottleneck operators during the inference process, reduces the overhead of activation value storage and temporary buffers for memory bottleneck operators by utilizing activation tiling, and reduces the computational load of computation time bottleneck operators by employing adaptive sparse computation. The specific details are as follows: 1) Analysis of the reasoning process This invention proposes a comprehensive analysis of the inference process of a video diffusion transformer, identifying memory overhead bottleneck operators and computation time bottleneck operators during the inference process: memory overhead bottleneck operators have activation value storage overhead exceeding the average level of all operators during inference, while computation time bottleneck operators account for the largest proportion of the total computation time. Furthermore, a roofline model is used to identify whether the operators are limited by memory transfer speed (Memory-bound) or computation hardware speed (Computation-bound).

[0016] Figure 1 The inference process analysis results of the typical video diffusion transformer model CogVideoX-1.5-5B (from the non-patent literature Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.) are presented: Figure 1 (a) reflects the storage overhead of activation values ​​for each operator. Figure 1 (b) shows the proportion of the total computation time for each operator. Figure 1 Figure (c) shows the position of each operator in the roofline model.

[0017] comprehensive Figure 1The results in (a) and (b) clearly show that the QK Normalization (QK Norm.) and Rotary Position Embedding (RoPE) operators in the Spatio-temporal 3D Full Attention module, and the input projection operator (Projection In, Proj. In), Gaussian Error LinearUnit (GELU), and output projection operator (Projection Out, Proj. Out) operators in the FeedForward module, exhibit significant activation value storage overhead. However, these operators account for less than 30% of the overall computation time. Figure 1 In (c), these operators also exhibit characteristics limited by memory transfer speed. Therefore, to address these identified memory bottlenecks, the activation tiling method proposed in this invention can be applied to reduce the memory overhead of activation storage and temporary buffers through feature slicing and multi-operator fusion.

[0018] The QKV Attention operator (QKV Attn.) in the spatiotemporal 3D full attention module has low activation value storage overhead, but its computation time accounts for more than 54% of the total inference time, exhibiting a characteristic limited by the computational hardware speed in the roofline model. To address this computational bottleneck, the adaptive sparse computation method proposed in this invention can be used. This technique is specifically designed to reduce computational load while maintaining inference accuracy.

[0019] 2) Activate tiling Activation tiling optimizes peak GPU memory overhead through two collaborative mechanisms: feature slicing, which slices the input features to reduce the size of activation values ​​during operator computation; and multi-operator fusion, which aggregates adjacent operator chains with similar arithmetic properties to reduce intermediate buffer allocation.

[0020] Feature slicing decomposes input features into smaller sub-features. The QK normalization operator and the rotation position encoding operator are calculated for each matrix of the feature; therefore, feature slicing cannot be performed on the last two dimensions of the feature. Instead, the integrity of the last two dimensions should be maintained (i.e., ensuring the integrity of all matrices in the feature), and slicing can be performed on other dimensions. Similarly, the input projection operator, Gaussian error linear unit operator, and output projection operator of the feedforward module are calculated for each row of the feature. Therefore, slicing cannot be performed on the last dimension of the feature, but slicing can be performed on other dimensions. This method of feature slicing based on operator-based feature calculation can reduce the size of activation values ​​without introducing additional computational errors.

[0021] After feature slicing, this invention fuses the QK normalization operator and the rotation position encoding operator into one operator group, and also fuses the three operators of the feedforward module into another operator group. Multi-operator fusion avoids reassembling the calculation results of all sub-features into full-size activation values ​​after a single operator's calculation is complete. Instead, it directly passes the sub-feature calculation results to adjacent operators to avoid large intermediate results, thereby reducing the GPU memory overhead of temporary buffers. Figure 2 The effect of applying activation tiling to the feedforward module is demonstrated. It can be seen that when the number of tiles is set to 2, the memory overhead of activation value storage and temporary buffer can be significantly reduced. As the number of tiles increases, the memory overhead will gradually decrease.

[0022] 3) Adaptive sparse computing Current sparse computation methods for spatiotemporal 3D full attention modules often affect inference accuracy. This invention proposes adaptive sparse computation to solve this problem, which can save computational resources and minimize the loss of video generation quality.

[0023] Attention maps (i.e., calculations) of different layers of the video diffusion transformer Analyzing the values ​​of the attention maps reveals that not all attention maps have a sparse shape. For example, for the CogVideoX-1.5-5B model, Figure 3 The comparison between attention maps with and without sparse shapes is shown. Forcing the addition of sparse masks to all attention maps, as in existing sparse computation methods, causes attention maps without sparse shapes to discard many important values, resulting in a significant loss of inference accuracy. Therefore, this invention proposes adaptive sparsity computation: adding sparse masks only to attention maps with sparse shapes, and not adding sparse masks to the remaining attention maps.

[0024] This invention proposes to identify layers suitable for sparse masks by comparing the P-value distributions of two sampling regions in an attention graph. Figure 3 This demonstrates a sampling region selection: region A is located in the upper left corner of the attention map, within the region where the sparse mask will retain computation; while region B is located in the upper right corner of the attention map, falling outside the sparse mask (i.e., region B will not be computed when the sparse mask is applied and will be directly assigned a value of 0). The core criterion for recognition is that if the sum of the P values ​​of region A is greater than or equal to the sum of the P values ​​of region A, then the region is considered to have a higher P value. A The sum of P values ​​in region B did not significantly exceed the sum of P values ​​in region B (Sum(P)). B If the attention map of this layer is not suitable for sparse masking, then the ratio Sum(P) is considered unsuitable. A ) / Sum(P B Sum(P) can be used as an effective measure of identification. A ) / Sum(P B When P < 10, the sum of P values ​​in the two regions is on the same order of magnitude, and it can be considered that the sum of P values ​​in region A does not significantly exceed the sum of P values ​​in region B.

[0025] This invention addresses the Sum(P) of the CogVideoX-1.5-5B model at different time steps, layers, and attention heads during the inference process. A ) / Sum(P B Statistical analysis was performed, and the results are as follows: Figure 4 As shown, it can be observed that whether an attention map is suitable for sparse masking is mainly determined by the layer number of the attention map, while the time step and attention head have almost no impact on the sparse pattern preference of the attention map. This finding leads to two key optimizations: 1) The sparse pattern evaluation of the attention map at a certain time step is still valid for subsequent time steps, so the attention maps of all layers can be identified in only one time step throughout the entire inference process; 2) Secondly, the sparse applicability of a certain layer is almost consistent across different attention heads, so the sparse pattern of the layer can be determined by a small number of randomly sampled attention heads through a "majority vote" scheme. For example, if 5 attention heads are randomly selected from 48 attention heads in a certain layer, the sparse preference of the attention map under these 5 attention heads is calculated. If the attention maps of more than 2 attention heads are not suitable for sparse masking, then the sparse pattern of that layer can be considered "not suitable for sparse masking".

[0026] The functions and effects of this invention are further illustrated and demonstrated through the following model reasoning experiments: 1) Experimental conditions To verify the universality of the invention, tests were conducted on two widely used video diffusion transformer models: HunyuanVideo (from non-patent literature Kong W, Tian Q, Zhang Z, et al. Hunyuanvideo: A systematic framework for large video generative models[J]. arXiv preprint arXiv:2412.03603, 2024.) and CogVideoX-1.5-5B (from non-patent literature Yang Z, Teng J, Zheng W, et al. Cogvideox: Text-to-video diffusion models with an expert transformer[J]. arXiv preprint arXiv:2408.06072, 2024.). After 50 time steps of denoising, HunyuanVideo generated 129 frames of video with a resolution of 768x1280, while CogVideoX generated 57 frames of video with a resolution of 720x1360. The first 12 time steps in all experiments did not use sparse computation methods, which is consistent with other sparse methods.

[0027] The inference accuracy of the original model output was evaluated using Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS, from non-patent literature Zhang R, Isola P, Efros AA, et al. The unreasonable effectiveness of deep features as a perceptual metric[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 586-595.). To simulate human perspective in evaluating video quality, the VisionReward metric (from non-patent literature Xu J, Huang Y, Cheng J, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation[J]. arXiv preprint arXiv:2412.21059, 2024.) was used for scoring.

[0028] The computation time and memory overhead of the model inference were measured on an NVIDIA A100-80GB GPU. The `enable_model_cpu_offload()`, `vae.enable_tiling()`, and `vae.enable_slicing()` settings were enabled for all experiments. This invention uses FlashInfer (from non-patent literature Ye Z, Chen L, Lai R, et al. Flashinfer: Efficient and customizable attention engine for llm inferenceserving[J]. arXiv preprint arXiv:2501.01005, 2025.) and FlashAttention-2 (from non-patent literature Dao T. Flashattention-2: Faster attention with better parallelism and work partitioning[J]. arXiv preprint arXiv:2307.08691, 2023.) as the inference backend, with all activation tiling numbers set to 12.

[0029] In the comparison of generation quality and GPU memory performance, tests were conducted using the sparse methods SVG (from non-patent literature Xi H, Yang S, Zhao Y, et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity[J]. arXiv preprint arXiv:2502.01776, 2025.) and Radial-attention (from non-patent literature Li X, Li M, Cai T, et al. Radial Attention: $O (n\log n) $ Sparse Attention with Energy Decay for Long Video Generation[J]. arXiv preprint arXiv:2506.19852, 2025.) as benchmarks. The sparse mask used in this application employs a Radial-attention mask, meaning that adaptive sparse computation requires determining whether all layers of the model are suitable for the Radial-attention mask. The precision-lossless weight offload method of the Python diffusers library (i.e., enabling the enable_sequential_cpu_offload() setting) and the INT8 weight quantization method of the Python TorchAO library were also evaluated. This application also additionally reports the results of combining the method of this invention with the TorchAO INT8 quantization method.

[0030] 2) Experimental Results ① Video generation quality testing and comparison The table below shows the video generation similarity and quality assessment results of the method of the present invention and the comparison method.

[0031]

[0032] In experiments at HunyuanVideo, the proposed method outperformed all other sparse methods in PSNR, SSIM, and LPIPS metrics, reflecting almost no degradation in the generated video compared to the original. In the VisionReward quality assessment metric, the proposed method also achieved a level close to the original model, outperforming other sparse methods. When combined with the TorchAO INT8 quantization method, the proposed method maintained good video similarity, achieving PSNR=26.01 and SSIM=0.878; simultaneously, VisionReward still outperformed other sparse methods.

[0033] Experiments on the smaller CogVideoX-1.5-5B model further confirm the effectiveness of the proposed method in maintaining inference quality. By dynamically selecting whether to apply a sparse mask through adaptive sparse computation, the proposed method significantly improves all similarity metrics compared to Radial-attention, while the VisionReward score is comparable to the original model. Although the accuracy loss of TorchAO INT8 quantization limits the improvement of similarity metrics (PSNR, SSIM, LPIPS) after combining with it, the VisionReward score of this combination remains at 0.0539. This reflects that the combination can maintain the evaluation of the generated video from a human perception perspective: the generated video still maintains clear visual effects, logical continuity, and semantic consistency with the prompts.

[0034] ② Measurement and Comparison of Model Inference Costs Figure 5 The experimental results for model inference overhead are presented. The X-axis represents model inference time, the Y-axis represents video generation quality, and the area of ​​the circle in the figure represents the GPU memory overhead of the corresponding method. Therefore, the smaller the area of ​​the circle near the top left corner of the image, the better the balance between generation quality, memory overhead, and computation time.

[0035] In the GPU memory overhead test conducted by HunyuanVideo, the method proposed in this invention reduced the peak memory usage by 28.9% from 49.84GB in the original model to 35.44GB, while other sparse methods maintained or increased memory usage. In terms of computation time, the proposed method reduced inference time by 40.5% compared to the original model (from 4246 seconds to 2525 seconds). In contrast, the `enable_sequential_cpu_offload()` method from the diffuses library increased inference time by 19.4% compared to the original model. Combined with the TorchAO INT8 quantization method, the proposed method reduced peak memory to 23.25 GB, enabling HunyuanVideo to be deployed and perform fast inference on consumer-grade GPUs such as the NVIDIA RTX 4090.

[0036] In the CogVideoX-1.5-5B experiment, the method proposed in this invention reduced GPU memory overhead by 15.8% and computation time by 11.1% compared to the original model. Combined with the TorchAO INT8 quantization method, the proposed method reduces inference memory overhead to 11.87 GB, thus enabling model deployment and fast inference on consumer-grade GPUs with 12 GB of memory (such as NVIDIA RTX 4070, NVIDIA RTX 3080 Ti, etc.).

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video generation model inference optimization method based on the collaboration of activation tiling and sparse computation, characterized in that, Includes the following steps: 1) Inference process analysis: Utilize the PyTorch Profiler library and PyTorch Snapshot function to analyze the memory overhead and computation time ratio of each operator in the model during the inference process. Use the roofline model in the NVIDIA Nsight Compute tool to identify whether the operator is limited by the memory transfer speed or the computing hardware speed, thereby obtaining the memory overhead bottleneck operator and the computation time bottleneck operator in the inference process. 2) Activation tiling: For operators with bottleneck memory overhead, an activation tiling method that combines feature slicing and multi-operator fusion is used to reduce the memory overhead of activation value storage and temporary buffers. 3) Adaptive sparse computation: Based on the characteristics of the attention map of the model layer, it adaptively selects whether to use a sparse mask to reduce the computational cost of model inference while maximizing the accuracy of inference.

2. The video generation model inference optimization method based on the collaboration of activation tiling and sparse computation as described in claim 1, characterized in that, In the inference process analysis, operators whose activation value storage overhead is higher than average and limited by the video memory transfer speed are identified as inference video memory bottlenecks, and operators that occupy the largest proportion of inference time and are limited by the computing hardware speed are identified as computing time bottlenecks.

3. The video generation model inference optimization method based on the collaboration of activation tiling and sparse computation as described in claim 1, characterized in that, The activation tiling method includes: first, feature slicing decomposes the input features of the QK normalization operator, the rotation position encoding operator, and the three operators of the feedforward module into smaller sub-features; then, after feature slicing, the QK normalization operator and the rotation position encoding operator are fused into one operator group, and the three operators of the feedforward module are fused into one operator group.

4. The video generation model inference optimization method based on the collaboration of activation tiling and sparse computation as described in claim 3, characterized in that, In the activation tiling, feature slices are performed based on the operator-calculated features, and adjacent operators with consistent calculated features are merged into an operator group; the QK normalization operator and the rotation position encoding operator maintain the integrity of the last two dimensions and perform segmentation in the dimensions other than the last two dimensions; the input projection operator, Gaussian error linear unit operator, and output projection operator of the feedforward module maintain the last dimension of the feature and perform segmentation in the dimensions other than the last dimension.

5. The video generation model inference optimization method based on the collaboration of activation tiling and sparse computation as described in claim 1, characterized in that, In adaptive sparse computation, layers suitable for sparse masks are identified by comparing the distribution of P-values ​​in two sampling regions of the attention map. Sparse masks are added only to layers with sparse shapes, thereby minimizing the loss of video generation quality. The core criterion for identification is: if the sum of the P values ​​of region A in the sparse mask does not exceed the sum of the P values ​​of region B not in the sparse mask, then the attention map of that layer is considered unsuitable for using a sparse mask.

Images