Quantization acceleration method for stable video diffusion model

Abstract

The invention relates to a quantization acceleration method for a stable video diffusion model, and the method comprises the steps: carrying out the comprehensive quantization acceleration design of a reasoning optimizer based on a U-Net network structure in three granularities: Kernel, Layer and Block, and carrying out the universal matrix multiplication quantization or operator fusion of different types of Kernel in a Kernel level; in Layer and Block levels, end-to-end schemes oriented to different scenes are provided through qualitative and quantitative analysis, and finally, the speed-up ratio of 1.63 is up to the maximum on the premise that the video precision is guaranteed. When the method is used for generating large model information in a diffusion type video, the loading and utilization rate of video data can be improved on the premise that the image and video precision is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of model data processing technology, and in particular to a quantization acceleration method for stable video diffusion models. Background Technology

[0002] In recent years, generative AI has made significant progress, with various types of large-scale models demonstrating their potential in different fields. Furthermore, with technological advancements, the modalities of generated content are constantly evolving, from text to images to video. By combining the traditional U-Net network with the modern Transformer module, the stable video diffusion model has become a leader among large-scale video generation models, renowned for its ability to generate high-quality images and videos. However, the long inference time of diffusion-type video generation models is unacceptable. Therefore, the technical problem this technology aims to solve is: how to reduce the inference time of the stable video diffusion model while maintaining a certain level of accuracy. The technical challenge lies in its complex model structure and the higher accuracy requirements of video compared to other information carriers. The designed technical solution revolves around the U-Net network, the main time-consuming component. From the perspective of its memory behavior, a hybrid precision quantization acceleration scheme is designed for different hardware specifications to accelerate the inference of the stable video diffusion model. Fixed-point quantization has been proven to be an effective method for acceleration and memory saving in traditional neural networks, improving memory utilization and computational efficiency by mapping complex data formats to simpler data formats. Summary of the Invention

[0003] To address the performance bottleneck of large-scale video generation models for diffusion-type videos, a quantization acceleration method for stable video diffusion models is proposed.

[0004] The technical solution of this invention is: a quantization acceleration method for a stable video diffusion model, based on the U-Net network structure, which performs comprehensive quantization acceleration design at the Kernel, Layer, and Block granularities of the inference optimizer; The innermost kernel layer is designed for acceleration. Non-general matrix multiplication operators are fused to reduce the total number of operators, thereby achieving tensor reuse and reducing switching overhead to accelerate inference. General matrix multiplication is accelerated by quantization. Efficient integer matrix multiplication operators are implemented by utilizing block matrix multiplication and multi-stage pipelines to accelerate computation. For the Layers outside the Kernel, Smoothquant is first used to pre-quantize and smooth the layers with the general matrix multiplication form of attention and feedforward. Then, based on the feature distribution of the activation tensors of different layers, the Mean-Min method is used to mitigate the influence of outliers. Mean refers to averaging the maximum absolute value of the activation tensors of all calibration images in the same layer at a given denoising time step, which is used to smooth out outliers between different images. Min refers to taking the minimum value obtained in the previous step as the scaling factor on each tensor in the denoising iteration dimension. At the same time, a few layers located in key positions and those related to time features are excluded from the quantization range to maintain video quality with almost no loss of speedup. For the Block layer outside the Layer layer, based on the different roles of different Blocks in the diffusion model, the signal-to-quantization-noise ratio (SQNR) is used as a metric for the quantization sensitivity of the Block to accurately identify the quantization impact of different Blocks.

[0005] Furthermore, the non-general matrix multiplication operator is merged and merged according to different strategies: two fusion operators are formed around the instance normalization operator and the sigmoid activation operator, respectively. The former reduces the switching overhead by putting multiple tensor transformation functions into one operator; the latter merges multiple operation operators into a composite operator through a composite function, which not only ensures complete mathematical equivalence, but also saves redundant operations between operations.

[0006] Furthermore, for general matrix multiplication, the task is split, and the number of small matrix blocks will be... SM In terms of thread blocks, the pipeline design overlaps the various stages of matrix operations, allowing different units of the SM to perform operations at the same time.

[0007] The beneficial effects of this invention are as follows: This invention provides a quantization acceleration method for stable video diffusion models, which improves the loading and utilization rate of video data while ensuring the accuracy of images and videos. Attached Figure Description

[0008] Figure 1 U-Net architecture diagram for stabilizing video diffusion model; Figure 2 This is the third-level optimization granularity diagram of the present invention; Figure 3 This is a graph showing the quantization speedup of the general matrix multiplication layer of this invention. Figure 4 This is a comparison chart of the quantification schemes of the present invention. Detailed Implementation

[0009] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0010] This invention presents a quantization acceleration method for stable video diffusion models: based on the U-Net network structure, a comprehensive quantization acceleration design is implemented at the Kernel, Layer, and Block levels for the inference optimizer. At the Kernel level, general matrix multiplication quantization or operator fusion is performed on different types of Kernels. At the Layer and Block levels, end-to-end solutions for different scenarios are proposed through qualitative and quantitative analysis. Ultimately, a speedup of up to 1.63 is achieved while ensuring video accuracy.

[0011] like Figure 1 The diagram shows the U-Net structure of the stable video diffusion model. Although the functions of each sampling block in the U-Net remain unchanged in the stable video diffusion model—downsampled blocks are still responsible for feature extraction and upsampled blocks are responsible for feature reconstruction—the structure of the U-Net in the stable video diffusion model is quite complex because the sampling blocks combine a Transformer (Spatio Temporal Transformer Block) and a ResNet (Spatio Temporal Res Block). Different layers have different effects in the algorithm, and different kernels also have vastly different computational behaviors. Uniform quantization can have a disastrous impact on accuracy.

[0012] This invention optimizes the U-Net structure of existing stable video diffusion models by dividing it into three levels of quantization compression granularity based on the U-Net network structure. For example... Figure 2 The diagram shows the three levels of optimization granularity, with the finest granularity being the Kernel layer. The analysis and optimization at this granularity are based on TensorRT (Inference Optimizer): operator fusion is performed on some non-general matrix multiplication operators to accelerate inference by reducing the total number of operators, achieving tensor reuse, and reducing switching overhead; general matrix multiplication is accelerated by quantization, and efficient integer matrix multiplication operators are achieved by utilizing block matrix multiplication and multi-level pipeline strategies to accelerate computation.

[0013] It should be noted that at the Kernel level, only inference acceleration is focused on without considering the impact on accuracy. This is because, although both the inference acceleration and accuracy loss brought by quantization originate from the lowest-level sum function, the essence of accuracy loss is that the bias caused by quantization makes the data in the model computation graph deviate from the original result of the algorithm. The concept of the algorithm is not reflected in the Kernel level, but rather in the Layer and Block levels above it. Therefore, the analysis and optimization of the accuracy impact starts from the Layer level. As mentioned above, only by calibrating the activation feature tensors can the accuracy of the quantization method after training be guaranteed. Although Smoothquantization has been proven to achieve good results in language models, it still has significant shortcomings in diffusion models, especially in video diffusion models optimized by the Kernel level of this invention. Therefore, this invention further mitigates the impact of outlier features by using the Mean-Min method based on the feature distribution of activation tensors in different layers. Then, based on their role in the algorithm, a few layers located in key positions and related to time features are excluded from the quantization range. Experiments show that after the above layer granularity optimization for accuracy, the accuracy of the stable video diffusion model generated by the quantization acceleration is significantly improved. At the block granularity, based on the fact that different blocks play different roles in the diffusion model, this invention uses the signal-to-quantization-noise ratio (SQNR) as a metric for block quantization sensitivity. This overcomes the problem that Vbench cannot finely distinguish the impact of single-sample block quantization on accuracy, thus enabling a comprehensive quantitative analysis of data on both accuracy and speed. This allows for efficient and relatively accurate identification of the quantization impact of different blocks, and a more reasonable design of end-to-end acceleration solutions.

[0014] Firstly, at the kernel granularity, within the TensorRT inference framework, non-general matrix multiplication operators reduce kernel function switching overhead through operator fusion and 8-bit integer matrix multiplication. Specifically, through the layer & operator fusion function, non-general matrix multiplication operators are fused and merged according to different strategies: two fused operators are formed around the instance normalization operator and the Sigmoid activation operator, respectively. The former reduces switching overhead by putting multiple tensor transformation functions into one operator; the latter uses composite functions to equivalently fuse multiple operations into one composite operator, ensuring complete mathematical equivalence and saving redundant operations between computations. To improve the parallel efficiency of GPUs for general matrix multiplication, a block-based matrix multiplication and multi-stage pipeline strategy are employed. The GPU's storage and computational structure differs from that of the CPU in that it has a large number of computational units (SMs), which can perform operations on data without dependencies with extremely high parallelism. Similar to the CPU, it utilizes a multi-stage storage structure to enhance the application of the principle of locality. Global memory shared by all thread blocks has a large capacity but low speed, while on-chip register files visible only within thread blocks are the fastest but have the smallest capacity. Therefore, on-chip shared memory is used as a buffer between the two. By processing the matrix in blocks, these two characteristics can be effectively utilized. This overcomes the problem in the stable video diffusion model U-Net where the activation and weights of general matrix multiplication are too large to fit into shared memory all at once, and also allows for the utilization of a large amount of... SM The resulting computational parallelism. (Regarding the output matrix) O Perform task splitting, so that each thread block has a shape of mtile× ntile matrix Otile ij The calculation, through in k The activation matrix is ​​obtained by moving along the axis. A With weight matrix W small block matrix on Atile ik and Wtile kj The shapes are respectively mtile×ktile and ktile×ntile The cumulative sum of the multiplication of these two small matrices is obtained. Otile ij Because the matrix multiplication scale in the stable video diffusion model is relatively large, the output matrix... O The number of small blocks that can be divided into matrices can SM The area is full, therefore calculation is performed. Otile ij At that time, each thread block performed ⌈ consecutively. kktile ⌉ Individual rather than single tile Matrix multiplication of a certain size, depending on the different operations performed during the operation, will result in each... tileLarge-scale matrix operations are divided into three stages: loading data from global memory to shared memory (LDGSTS), loading data from shared memory to registers (LDSM), and performing block-based integer matrix multiplication (IMMA). Since there are no data dependencies between consecutive operations within a thread block, the various stages of matrix operations can be overlapped at the thread block level through pipelined design. This allows different units within the SM to execute operations simultaneously, maximizing hardware utilization.

[0015] Secondly, at the layer granularity, Smoothquant is used to pre-quantize and smooth layers with general matrix multiplication forms such as attention and feedforward. Smoothquant significantly smooths the amplitude between channels, making the model easier to quantize and improving inference speed while maintaining accuracy. Min-Mean calibration is applied to convolutional layers that cannot be directly represented as linear multiplication to reduce the accuracy loss caused by outliers. This is because Smoothquant is a pre-quantization strategy, requiring the operations of relevant layers to be expressed as general matrix multiplication forms before the general matrix multiplication. In the stable video diffusion model U-Net, linear mapping layers of attention and feedforward layers meet this requirement, while convolutional layers cannot be directly represented in this form before the general matrix multiplication. Therefore, other methods are needed to mitigate the impact of outliers. As mentioned above, activation tensors at the same location will produce calibrated data that differ in two dimensions: the difference between different calibration images and the difference between different denoising iterations of the same image. Severe outlier features between denoising iterations will only appear in the last few denoising iterations. The degree of shift in outlier values ​​caused by different calibration images is smaller than that of the former. Mean refers to averaging the absolute maximum values ​​of the activation tensors corresponding to all calibration images at a given denoising time step within the same layer. This is used to smooth out outliers between different images because the degree of deviation of outliers in this dimension is relatively small and the distribution is relatively uniform. Averaging mitigates the impact of values ​​with higher deviations in this dimension. Min refers to taking the minimum value obtained in the previous step as the scaling factor per tensor in the denoising iteration dimension. As observed above, the outliers in the last few steps of the denoising iteration are too abnormal and are discarded, while the distribution in most of the remaining iterations is relatively smoother. Since the Mean operation has already preserved the impact of values ​​with higher deviations by averaging, the minimum value in this dimension is taken as the scaling factor per tensor. At the same time, for the temporal feature-related layer and the feature extraction and reconstruction layer, because their computational scale is small but their impact on the video generation effect is large, they are excluded from the quantization range to maintain video quality with almost no loss of speedup.

[0016] Finally, at the block level, sampling blocks that are more sensitive to low-order operations are identified and restored to higher precision to prevent errors from being amplified during generation. Through the collaborative design at these three levels of granularity, a hybrid precision quantization scheme that balances inference speed and generation quality is achieved, providing a practical solution for the actual deployment of stable video diffusion models.

[0017] like Figure 3 The graph showing the quantization speedup of the general matrix multiplication layer in the Nvidia A100 demonstrates that while TensorRT's half-precision floating-point operations (orange) show some improvement over PyTorch (blue line), the improvement is not significant, and some layers are even slower. Thanks to operator fusion and optimizations at the inference framework level, a single U-Net inference (gray) achieves 27% higher performance than PyTorch's half-precision matrix multiplication. Furthermore, by implementing 8-bit integer operations for general matrix multiplication, the data bit width is further halved compared to half-precision floating-point operations, thus improving bandwidth utilization. The GPU's advantage in processing integer data further accelerates the computation, resulting in a significant improvement in the speedup of operators at each layer, reaching a maximum speedup of 3.23. The overall inference speedup of U-Net is further improved by 21% on top of TensorRT's half-precision floating-point operations, and a single U-Net inference achieves a speedup of 1.75 compared to PyTorch's half-precision floating-point operations.

[0018] Ultimately Figure 4 The comparison chart of quantization schemes shown illustrates the end-to-end acceleration schemes based on quantization compression. A detailed comparison was conducted at the Kernel, Layer, and Block granularities. From the performance advantages of fixed-point integer matrix multiplication to the selection of calibration methods, from the comprehensive analysis of the impact of the quantization sensitive layer on accuracy and speed to the impact of different sampling blocks on accuracy, end-to-end inference schemes for different scenarios were designed and analyzed. The accuracy degradation can be controlled within 0.5%, while the speedup ratio can reach up to 1.63.

[0019] The embodiments described above merely illustrate specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A quantization acceleration method for stable video diffusion models, characterized in that, Based on the U-Net network architecture, comprehensive quantization acceleration design is implemented at the Kernel, Layer, and Block levels of the inference optimizer. Acceleration is achieved in the innermost Kernel layer by fusing non-general matrix multiplication operators, reducing the total number of operators, enabling tensor reuse, and lowering switching overhead to accelerate inference. General matrix multiplication is accelerated by quantization using block matrix multiplication and a multi-stage pipeline to implement efficient integer matrix multiplication operators. For the Layer layers outside the Kernel, Smoothquantization is first applied to the attention and feedforward general matrix multiplication layers for pre-smoothing, and then, based on the characteristic distribution of the activation tensors of different layers, Mean-Min is further applied for smoothing. The impact of outlier features is addressed by: Mean, which refers to averaging the absolute maximum values ​​of the activation tensors corresponding to all calibrated images at a given denoising time step within the same layer, used to smooth out outliers between different images; and Min, which refers to taking the minimum value obtained in the previous step as the scaling factor per tensor in the denoising iteration dimension. Simultaneously, a few layers located in critical positions and those with temporal features are excluded from the quantization range to maintain video quality with almost no loss in speedup. For Block layers outside the Layer layer, based on the different roles of different Blocks in the diffusion model, the Signal-to-Quantization-Noise Ratio (SQNR) is used as a metric for Block quantization sensitivity to accurately identify the quantization impact of different Blocks.

2. The quantization acceleration method for a stable video diffusion model according to claim 1, characterized in that, The non-general matrix multiplication operator is merged and merged according to different strategies: two fusion operators are formed around the instance normalization operator and the Sigmoid activation operator, respectively. The former reduces the switching overhead by putting multiple tensor transformation functions into one operator; the latter merges multiple operation operators into a composite operator through a composite function, which not only ensures complete mathematical equivalence, but also saves redundant operations between operations.

3. The quantization acceleration method for a stable video diffusion model according to claim 2, characterized in that, For general matrix multiplication, task decomposition is performed, and the number of small matrix blocks will be... SM In terms of thread blocks, the pipeline design overlaps the various stages of matrix operations, allowing different units of the SM to perform operations at the same time.

Images