Diffusion model compression and acceleration method based on non-data quantization
Through quantitative compression and dynamic sampling data on the time step of the pre -training diffusion model, quantify the sampling data on the time step, solve the problem of slow and large number of parameters to generate high -quality images of the diffusion model. High -quality image output.
Patent Information
- Application Number
- CN202411168184.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2025-01-24
AI Technical Summary
The existing diffusion model is slow in generating high-quality images and has huge parameters, resulting in large memory consumption and reduced inference speed, especially on large-scale data sets and desktop-level graphics cards.
By quantizing and compressing the model parameters of the convolution layer and activation layer in the pre-trained diffusion model, and using the method of dynamic sampling at time steps to obtain sampling data, perform quantitative calibration, and obtain the final quantitatively calibrated diffusion model. .
The diffusion model is effective without dependence on the original data set, which has effective quantitative calibration, improves the speed of reasoning, and can output higher -quality images, so that the desktop -level graphics card can support the training of large models and the deployment of downstream application tasks.
Smart Images
Figure CN119359572A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence computer technology, and in particular to a diffusion model compression and acceleration method based on data-free quantization. Background Art
[0002] At present, the diffusion model can generate high-quality realistic images, and is therefore widely used in various generation tasks, such as the field of image generation. However, the main shortcomings are: 1) the model needs to go through a large number of iterations in the process of generating high-quality images, so the generation speed is relatively slow; 2) the current model has a huge number of parameters, which usually consumes a large amount of memory and leads to a decrease in inference speed. In response to the above shortcomings, the current DDIM, ODE, SDE and other methods mainly start from the sampler, and use skipping and distillation methods to reduce the number of iterations of the model sampling. This method can sample more realistic images in a shorter time. In response to the overload problem caused by excessive model parameters, in some fields, quantization methods are currently widely used to compress model parameters, that is, low-bit parameters such as Int8 are used to replace FP32 parameters, thereby reducing model parameters and improving inference speed. Quantization methods for diffusion models have gradually attracted attention. However, existing methods usually rely on calibration data. Although the calibration data only requires a small amount of the original data set, it is extremely difficult to use large-scale data sets such as LAION-400M and LAION-5B on current general-purpose desktop graphics cards. The main reasons for this approach are: first, the current data set size is still very large, so it is difficult to implement; second, the performance of desktop graphics cards is difficult to support the training of large models, which is not conducive to the quantization of the model in downstream application tasks after release. Summary of the invention
[0003] The present invention discloses a diffusion model compression and acceleration method based on data-free quantization to overcome the above technical problems.
[0004] In order to achieve the above object, the technical solution of the present invention is:
[0005] A diffusion model compression and acceleration method based on data-free quantization includes the following steps:
[0006] S1: quantize and compress the model parameters in the convolution layer and activation layer of the pre-trained diffusion model to obtain a quantized and compressed diffusion model;
[0007] S2: obtaining sampling data at the time step through the pre-trained diffusion model based on a method of dynamic sampling at the time step according to the sampler and Gaussian noise, i.e., calibration data for calibrating the quantized compressed diffusion model at the time step;
[0008] S3: obtaining a final quantized and calibrated diffusion model according to the sampled data at the time step and the quantized and compressed diffusion model;
[0009] S4: According to the final quantitatively calibrated diffusion model and the actual data set in the specific business scenario, an inferred image based on the specific business scenario is obtained to complete the inference process of the actual data set in the specific business scenario.
[0010] Furthermore, the method for performing dynamic sampling on the time step is as follows:
[0011] S21: inputting random Gaussian noise, time step ΔT, the total number of time steps and sampling conditions into the pre-trained diffusion model;
[0012] The sampling condition is the image data or text prompt words used in a specific business scenario;
[0013] S22: sequentially at time t, time t+ΔT, time t+2ΔT, …, time t+n·ΔT, …, time t+N·ΔT, sampling is performed in the pre-trained diffusion model by the sampler to obtain sampling data at time t, time t+ΔT, …, time t+n·ΔT, …, time t+N·ΔT;
[0014] Wherein, t represents the initial sampling time of the sampler; n represents the index number of the sampling period, n=0, 1, ..., N; N represents the number of the period of the last sampling.
[0015] Furthermore, the method for quantizing and compressing the model parameters is as follows:
[0016]
[0017] Where: x int represents the model parameters after quantization; x represents the model parameters before quantization; s represents the scale factor; [[·]] represents the rounding symbol; clamp(·) represents the truncation operation function; o represents the zero point; b min Indicates the lower bound of parameter quantization and clipping; b max Indicates the upper bound for quantizing and clipping parameters.
[0018] Furthermore, the method used to obtain the final quantitatively calibrated diffusion model is as follows:
[0019] S31: training the quantized and compressed diffusion model according to the sampling data at time t and the loss function of the model to obtain the diffusion model after the first quantization calibration;
[0020] S32: training the diffusion model after the first quantization calibration according to the sampling data at time t+ΔT and the loss function of the model to obtain the diffusion model after the second quantization calibration;
[0021] S33: training the diffusion model after the second, ..., n, ..., N-1th quantization calibration according to the sampling data at time t+2·ΔT, ..., t+n·ΔT, ..., t+(N-1)·ΔT and the loss function of the model, to obtain the diffusion model after the third, ..., n+1, ..., Nth quantization calibration;
[0022] S34: training the diffusion model after the Nth quantization calibration according to the sampling data at time t+N·ΔT and the loss function of the model to obtain the final diffusion model after the quantization calibration.
[0023] Furthermore, the loss function of the model is:
[0024]
[0025] Where: L quant represents the value of the loss function; X sim is the dequantized tensor; Metric is the metric function; A Tensor representing full-precision features.
[0026] Beneficial effects: The diffusion model compression and acceleration method based on data-free quantization of the present invention obtains a quantized compressed diffusion model by quantizing and compressing the model parameters in the pre-trained diffusion model; at the same time, according to the sampler and the method of dynamic sampling on the time step, the sampling data on the time step is obtained on the pre-trained diffusion model, and then combined with the quantized compressed diffusion model, the final quantized calibrated diffusion model is obtained. Finally, according to the actual data or text prompt conditions in the specific business scenario, it is input into the final quantized calibrated diffusion model to obtain the image after reasoning based on the specific business scenario, and the reasoning process of the actual data set in the specific business scenario is completed. The present invention does not directly use the original data set for calibration, but the use of a data set constructed by Gaussian noise sampling for calibration can also achieve an effect equivalent to the original data set sampling calibration. Therefore, the present invention does not rely on the original data to calibrate the quantized compressed diffusion model, so as to obtain the final quantized calibrated diffusion model, that is, the accelerated reasoning system. The present invention calibrates the model by sampling the data obtained in the pre-trained diffusion model, so that the quantized model can obtain the same ability as the original model, making the reasoning speed faster. At the same time, the quantized calibrated diffusion model obtained by calibrating the sampled data obtained by dynamic sampling at the time step can output higher quality images. The problem of difficult implementation due to the huge scale of the data set is solved, so that the performance of the desktop-level graphics card can easily support the training of large models, which is more conducive to the deployment of the model in downstream application tasks after release. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0028] Figure 1 It is a flow chart of the diffusion model compression and acceleration method based on data-free quantization of the present invention;
[0029] Figure 2 It is a flow chart of a diffusion model compression and acceleration method based on data-free quantization in an embodiment of the present invention;
[0030] Figure 3 It is a framework diagram of the specific implementation process of the diffusion model compression and acceleration method based on data-free quantization in an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] This embodiment introduces a diffusion model compression and acceleration method based on data-free quantization, including the following steps: Figure 1-3 shown.
[0033] S1: quantize and compress the model parameters in the convolution layer and activation layer of the pre-trained diffusion model to obtain a quantized and compressed diffusion model;
[0034] Preferably, the method for quantizing and compressing the model parameters is as follows:
[0035]
[0036] Where: x int represents the model parameters after quantization; x represents the model parameters before quantization; s represents the scale factor; [[·]] represents the rounding symbol, that is, the scaled parameters are rounded off; clamp(·) represents the truncation function, which can control the range of the parameters within b min and b max between; o represents zero point; b min Indicates the lower bound of parameter quantization and clipping; b max Indicates the upper bound for quantizing and clipping parameters.
[0037] Specifically, a pre-trained model refers to a well-trained diffusion model obtained by training on a large dataset, which can be used in other downstream tasks. By fine-tuning it, images in specific fields, such as clothing, animals, vehicles, etc., can be trained more quickly.
[0038] It is a trained model that can meet the needs of different tasks, such as tasks that require the generation of clothing category images, tasks that generate animation images, etc. The pre-trained diffusion model is the main body of compression, which includes basic models such as stable diffusion model, latent diffusion model, pixel space diffusion model (DDPM) and related variants, that is, other improved architectures based on the diffusion model. The model is sampled using random Gaussian noise e and a method of dynamic sampling in time steps and sampling conditions c as input. According to the specified sampling conditions (the conditions are mainly prompt words and the like), through iteration, the model can restore and generate image x0 from the noise during the reverse denoising process. And the sampled data is used to construct a calibration set.
[0039] Specifically, the model parameters in the pre-trained diffusion model are floating-point 32-bit data (Float32), which occupies a large amount of memory. Therefore, this embodiment compresses the model parameters in the pre-trained diffusion model and converts the model parameters from floating-point 32-bit data (Float32) to integer 8-bit data (Int8). After the model parameters in the pre-trained diffusion model are quantized and compressed, the memory occupancy of the model parameters is relatively low, which can speed up the inference speed. After the model parameters are quantized and compressed, a quantized and compressed diffusion model can be obtained. Specifically, in this embodiment, the weights in the convolutional layer and the activation layer are mainly quantized and compressed. During the compression process, the model parameters are quantized by rounding (Adaptive rounding), which converts the floating-point parameters x FP32 Quantized to integer value x Int8 , where the quantized parameters include the scale factor s and the zero point o. After quantization, the parameters of the model are controlled within [b min ,b max ], b min and b max Respectively represent the lower and upper bounds of parameter quantization and pruning. Realize the reconstruction of the encoder and decoder in the UNet network in the latent space.
[0040] S2: according to the sampler and Gaussian noise, based on a method of dynamic sampling at the time step, through the pre-trained diffusion model, obtaining sampling data at the time step, that is, calibration data for calibrating the quantized compressed diffusion model at the time step;
[0041] Preferably, the method for performing dynamic sampling on the time step is as follows:
[0042] S21: inputting random Gaussian noise, time step ΔT, the total number of time steps and sampling conditions into the pre-trained diffusion model; the sampling conditions are image data or text prompt conditions used in a specific business scenario;
[0043] Specifically, the data set used in a specific business scenario generally refers to the data set used in a vertical field, which is determined by the specific business scenario. For example, the diffusion model used in clothing generation requires a large amount of clothing data for training and sampling. Similarly, the application of medical image generation also requires training and sampling on a large number of medical images.
[0044] S22: At time t, time t+ΔT, time t+2ΔT, …, time t+n·ΔT, …, time t+N·ΔT, the sampler performs sampling in the pre-trained diffusion model to obtain sampling data at time t, time t+ΔT, …, time t+n·ΔT, …, time t+N·ΔT.
[0045] S3: Obtain a final quantized and calibrated diffusion model according to the sampled data at the time step, the quantized and compressed diffusion model, and the loss function of the model.
[0046] Specifically, if the inference results are directly sampled through the quantized and compressed diffusion model, the quality of the generated samples will be reduced. Therefore, it is necessary to calibrate the quantized pre-trained diffusion model through a calibration data set. Specifically, random Gaussian noise e is input into the pre-trained diffusion model after loading, and sampling conditions c are input at the same time, where condition c is mainly a prompt word or image such as text, and dynamic sampling is performed on the time step through the sampler to obtain sampling data at time t, time t+ΔT, ..., time t+n·ΔT, ..., time t+N·ΔT in turn; obtain sampling data based on the sampling conditions at each time step. These sampling data are sample data that are close to the original data and can be used for specific tasks, and are used as calibration data for constructing a calibration set.
[0047] Specifically, the sampler may be a DDIM sampler, a PLMS sampler or any other existing sampler, which samples from a pre-trained diffusion model according to different time steps and Gaussian noise to obtain a sample set that can be used for a calibration set.
[0048] In this embodiment, the calibration set used to calibrate the quantized and compressed diffusion model does not use the original data set to construct the calibration set, but is obtained by sampling from a pre-trained diffusion model through a sampler during the training process. Random Gaussian noise, sampling steps, and sampling conditions are input into the pre-trained diffusion model, and the output of the model is the constructed calibration set.
[0049] Specifically, this embodiment minimizes the loss and updates the quantized model by calculating the reconstruction loss of the data before and after calibration. The reconstruction loss is calculated by using the mean squared error (MSE) to calculate the difference loss in the output features of the model before and after quantization. The calibration method mainly uses the calibration samples obtained by pre-sampling as the input of the pre-quantization / post-quantization model at the same time, and then calculates the difference before and after quantization of each layer.
[0050] Preferably, the method used to obtain the final quantitatively calibrated diffusion model is as follows:
[0051] S31: training the quantized and compressed diffusion model according to the sampling data at time t and the loss function of the model to obtain the diffusion model after the first quantization calibration;
[0052] Preferably, the loss function of the model is:
[0053] Specifically, in this embodiment, the quantized compressed diffusion model is calibrated, including calibration of model weights and activation-related parameters, by calculating the difference between the quantized model and the full-precision model: L2 loss, L1 loss or KL divergence:
[0054]
[0055] Where: L quant represents the value of the loss function; X sim It is the dequantized tensor; Metric is the metric function, which can be Mean Square Error (MSE), cosine distance, Manhattan distance (L1 distance) and KL divergence, which is used to evaluate the distance between the full-precision tensor and the quantized tensor. A tensor representing full-precision features, that is, a 32-bit floating point (float32) feature tensor.
[0056] Specifically, is the input, X sim It is the output, which is the parameter after quantization and then dequantization. Because there is no Int8 test environment in the training environment, this is actually a simulation test.
[0057] S32: training the diffusion model after the first quantization calibration according to the sampling data at time t+ΔT and the loss function of the model to obtain the diffusion model after the second quantization calibration;
[0058] S33: training the diffusion model after the second, ..., n, ..., N-1th quantization calibration according to the sampling data at time t+2·ΔT, ..., t+n·ΔT, ..., t+(N-1)·ΔT and the loss function of the model, to obtain the diffusion model after the third, ..., n+1, ..., Nth quantization calibration;
[0059] S34: training the diffusion model after the Nth quantization calibration according to the sampling data at time t+N·ΔT and the loss function of the model to obtain the final diffusion model after the quantization calibration.
[0060] The method used to train the diffusion model is a conventional technique in the field and will not be described in detail here.
[0061] S4: According to the final quantitatively calibrated diffusion model and the actual data set in the specific business scenario, after the actual data set in the business scenario is input into the final quantitatively calibrated diffusion model, the inferred image based on the specific business scenario can be obtained to complete the inference process of the actual data set in the specific business scenario (that is, the actual image data or the actual text prompt words used in the specific business scenario).
[0062] Specifically, this embodiment uses the preprocessed diffusion model stablediffusion and uses the data set in the specific business scenario to quantize and compress it. The preprocessed diffusion model mainly includes the VAE network of the encoder and decoder in the first stage. The VAE network can reduce the feature dimension to a smaller dimension latent vector z for the calculation of the latent space UNet network to obtain a quantized and compressed diffusion model, which greatly reduces the computational complexity of the pixel space. Next, use the DDIM or PLMS sampler to input the sampling condition c, the time step ΔT and the random Gaussian noise e. According to the time step ΔT, dynamically sample N+1 sample data with an interval of ΔT from the fine-tuned diffusion model, that is, the pre-trained diffusion model, thereby obtaining a combination of the N+1 sample data [x0, x1, ..., x N ]. Where x0 is the sampling data at time t, x1 is the sampling data at time t+ΔT, and x Nis the sampling data at time t+N·ΔT. Then, according to the sampling data at each time step, the diffusion model after quantization compression is continuously calibrated. Through calculation, the value of each layer of extracted features in the network during forward diffusion and reverse denoising is continuously reconstructed, and the difference loss of features in the full-precision model and the quantized model is calculated using the loss function of the model to obtain the final quantized calibrated diffusion model. After quantization compression, the quantized diffusion model has smaller parameters and faster inference speed. The quantized calibrated diffusion model obtained after calibrating with the data at the time step can output higher quality images.
[0063] Specifically, in this embodiment, the main parameters in the model training process are Gaussian noise e, time step ΔT, initial sampling time t, and sampling condition c. When the training loss is within a certain range (loss threshold), such as 0.4±0.01, the accuracy gradually converges and the network training ends.
[0064] Specifically, the method of this embodiment can also be applied to other diffusion models, such as pixel-space diffusion models, such as DDIM, DDPM and latent-space diffusion models, such as Latent Diffusion. The specific process will not be described in detail again.
[0065] In this embodiment, the parameters of the model determined by the experimental results can be used to select a set of parameters with better accuracy by testing different parameter combinations. In actual testing, the above parameters can be appropriately adjusted according to needs to achieve the purpose of the present invention.
[0066] In this example, the LSUN-Bedroom and LSUN-Churches datasets were selected as implementation data, and FID (Frechet Inception Distance score) and IS (Inception Score) were used as evaluation indicators. The implementation results are shown in the following table:
[0067] Table 1. Experimental results on the LSUN-Bedroom256x256 dataset (w means calibration is used, w / o means no calibration is used)
[0068]
[0069]
[0070] Table 2. Experimental results on the LSUN-Churches256x256 dataset.
[0071] Calib Bits(W / A) IS FID Baseline W32A32 10.1 4.2 w / o W4A8 3.6 29.5 w W4A8 7.1 6.2 w / o W8A8 5.3 15.9 w W8A8 9.8 4.8 w / o W8A4 8.6 127.5 w W8A4 14.6 67.8
[0072] As shown in Table 1 above, the original model achieves the FID values of 3.2 and 4.2 on the datasets Bedroom256 and Churches256 under full precision (FP32), i.e., W32A32. When the activation is 8-bit and the weight is 8-bit, the FID evaluation is 17.1, and when we calibrate the model with the calibration set, the FID of the sampled data is 4.9; when we further directly quantize the model weights to 4-bit and the activation to 8-bit, the FID evaluation of the model is 21.2, and after calibrating the model with the calibration set, the sampling result is 5.8. This shows that our calibration process can significantly improve the effect of the model. Similarly, on the LSUN-Churches dataset, we can also see that our method can significantly improve the loss caused by the quantized model. At the same time, by comparison, we can also find that the effect of W4A8 quantization is worse than that of W8A8 quantization. This shows that the use of a model with a low bit number will cause the performance of the model to decline, especially the quantization of the activation weight has a greater impact on the calibration of the model.
[0073] The present embodiment is a diffusion model compression and acceleration method based on data-free quantization, which obtains a quantized compressed diffusion model by quantizing and compressing the model parameters in the pre-trained diffusion model; at the same time, according to the sampler and the method of dynamic sampling on the time step, the sampling data on the time step is obtained on the pre-trained diffusion model, and then combined with the quantized compressed diffusion model, the final quantized calibrated diffusion model is obtained. Finally, according to the actual data set in the specific business scenario, it is input into the final quantized calibrated diffusion model to obtain the image after reasoning based on the specific business scenario, and the reasoning process of the actual data set in the specific business scenario is completed. The present invention does not directly use the original data set for calibration, but the use of a data set constructed by Gaussian noise sampling for calibration can also achieve an effect equivalent to the original data set sampling calibration. Therefore, the present invention does not rely on the original data to calibrate the quantized compressed diffusion model to obtain the final quantized calibrated diffusion model, that is, the accelerated reasoning system. The present invention calibrates the model by sampling the data obtained in the pre-trained diffusion model, that is, the original model, so that the quantized model can obtain the same ability as the original model, making the reasoning speed faster. At the same time, the quantized calibrated diffusion model obtained by calibrating the sampling data obtained by dynamic sampling at the time step can output higher quality images.
[0074] Since the present embodiment does not apply the original operation data, the training and fine-tuning of the diffusion model do not rely on the original data, thus avoiding the leakage of the original data during the data transmission process and protecting the privacy of the user.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A diffusion model compression and acceleration method based on data-free quantization, characterized in that: The steps include: S1: quantize and compress the model parameters in the convolution layer and activation layer of the pre-trained diffusion model to obtain a quantized and compressed diffusion model; S2: obtaining sampling data at the time step through the pre-trained diffusion model based on a method of dynamic sampling at the time step according to the sampler and Gaussian noise, i.e., calibration data for calibrating the quantized compressed diffusion model at the time step; S3: obtaining a final quantized and calibrated diffusion model according to the sampled data at the time step and the quantized and compressed diffusion model; S4: According to the final quantitatively calibrated diffusion model and the actual data set in the specific business scenario, an inferred image based on the specific business scenario is obtained to complete the inference process of the actual image data or the actual text prompt words in the specific business scenario.
2. The method for compression and acceleration of a diffusion model based on data-free quantization according to claim 1, characterized in that: The method for dynamic sampling on the time step is as follows: S21: inputting random Gaussian noise, time step ΔT, the total number of time steps and sampling conditions into the pre-trained diffusion model; The sampling condition is the image data or text prompt words used in a specific business scenario; S22: sequentially at time t, time t+ΔT, time t+2ΔT, …, time t+n·ΔT, …, time t+N·ΔT, sampling is performed in the pre-trained diffusion model by the sampler to obtain sampling data at time t, time t+ΔT, …, time t+n·ΔT, …, time t+N·ΔT; Wherein, t represents the initial sampling time of the sampler; n represents the index number of the sampling period, n=0, 1, ..., N; N represents the number of the period of the last sampling.
3. The method for compression and acceleration of a diffusion model based on data-free quantization according to claim 1, characterized in that: The method for quantizing and compressing model parameters is as follows: Where: x int represents the model parameters after quantization; x represents the model parameters before quantization; s represents the scale factor; Indicates the rounding symbol; clamp(·) indicates the truncation function; o indicates the zero point; b min Indicates the lower bound of parameter quantization and clipping; b max Indicates the upper bound for quantizing and clipping parameters.
4. The method for compression and acceleration of a diffusion model based on data-free quantization according to claim 2, characterized in that: The method used to obtain the final quantitatively calibrated diffusion model is as follows: S31: training the quantized and compressed diffusion model according to the sampling data at time t and the loss function of the model to obtain the diffusion model after the first quantization calibration; S32: training the diffusion model after the first quantization calibration according to the sampling data at time t+ΔT and the loss function of the model to obtain the diffusion model after the second quantization calibration; S33: training the diffusion model after the second, ..., n, ..., N-1th quantization calibration according to the sampling data at time t+2·ΔT, ..., t+n·ΔT, ..., t+(N-1)·ΔT and the loss function of the model, to obtain the diffusion model after the third, ..., n+1, ..., Nth quantization calibration; S34: training the diffusion model after the Nth quantization calibration according to the sampling data at time t+N·ΔT and the loss function of the model to obtain the final diffusion model after the quantization calibration.
5. The method for compression and acceleration of a diffusion model based on data-free quantization according to claim 4, characterized in that: The loss function of the model is: Where: L quant represents the value of the loss function; X sim is the dequantized tensor; Metric is the metric function; A Tensor representing full-precision features.