Post-training quantization method suitable for autoregressive mask generation model

Abstract

The invention provides a post-training quantization method, namely Q-MAR (Quantization for Masked Autoregressive Models), suitable for an autoregressive mask generation model, and the method innovatively designs two modules: a hierarchical clustering decoupling module (HCD) is used for identifying and decoupling a channel containing a maximum abnormal value (EOs) in a Transform to realize separation quantization of a normal value and an abnormal value, and a quantization module is used for identifying and decoupling a channel containing a maximum abnormal value (EOs) in the Transform to realize quantization of the normal value and the abnormal value. A scaling recalibration module (SR) is used for performing dynamic scaling control on activation drift in a diffusion network, stabilizing an activation range of an early time step, and inhibiting model collapse caused by amplification of an abnormal value (AAs) due to a quantization error. According to the Q-MAR, the MAR model can be stably compressed to 8-bit precision (W8A8) on the premise that the model does not need to be retrained, the storage and computing resource consumption is remarkably reduced while the generation quality is kept, and a feasible technical support is provided for deployment of the MAR model on resource-constrained equipment such as a mobile terminal and an edge terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to a post-training method of a generation model, in particular to a post-training quantization method suitable for an autoregressive mask generation model, and belongs to the field of generative artificial intelligence. BACKGROUND

[0002] With the wide application of diffusion models and Transformer architecture in generation tasks, great breakthroughs have been made in generative artificial intelligence in recent years. In particular, in the field of high-quality image generation, masked autoregressive models (MAR) have gradually become one of the mainstream technologies. MAR models combine autoregressive mechanisms and diffusion modeling mechanisms, which not only improve the generation quality but also enhance the modeling ability of the model for multi-token probability distribution. Specifically, it generates a conditional vector step by step through an autoregressive Transformer module, guiding the diffusion network to gradually restore the image tokens. This structure breaks the limitations of traditional autoregressive models, supports continuous space modeling, and has stronger modeling flexibility and representation ability.

[0003] However, MAR models also bring extremely high computational and storage costs. On the one hand, the token-by-token inference method means that the entire generation process must perform hundreds of autoregressive predictions and diffusion denoising, each of which involves a large number of matrix multiplications and nonlinear transformations. On the other hand, the model structure is large, especially in large variants (such as MAR-L, MAR-H) which contain more linear layers, making the deployment cost further skyrocket. Therefore, how to compress MAR models and reduce their inference resource overhead has become a core problem in the process of deploying generation models.

[0004] Under this background, post-training quantization (PTQ) has become a solution that has attracted much attention. This method compresses model parameters and activations from 32-bit floating-point to 8-bit or lower integer form after the model training is completed, achieving model compression and inference acceleration. Compared with traditional quantization-aware training, PTQ does not need to retrain the model and can be completed with only a small amount of calibration data, making it suitable for scenarios where time is tight and resources are limited before deployment, and thus has been widely used in the industry.

[0005] The PTQ method has made significant progress in the Transformer and Diffusion model. For example, SmoothQuant proposes a smoothing mechanism between activations and weights, which alleviates the quantization error caused by the extremely uneven distribution of activations in the Transformer. PTQ4DM addresses the changes in activation distribution at different times in the Diffusion model and uses a reconstruction-based quantization strategy. RepQ-ViT proposes a decoupling technique for calibration and inference process for LayerNorm and post-softmax to improve the quantization accuracy in the ViT structure. However, the above methods are based on traditional Transformer or DDPM structure design and do not consider the unique double nested structure and high iteration inference mode of MAR model. When applied to MAR model, the effect is greatly reduced.

[0006] Based on the above shortcomings, the present application proposes a post-training quantization method (Q-MAR) designed for the structure characteristics of MAR model, which realizes the compression of MAR to W8A8 precision without retraining, and maintains the image generation quality and stability, filling the method gap in this direction. SUMMARY

[0007] The post-training quantization method for autoregressive mask generation model introduced in the present application Hierarchical Cluster Decoupling (HCD) and Scaling Recalibration (SR) respectively solve the two problems of Excessive Outliers (EOs) and Amplified Anomalies (AAs) from the structure level and distribution alignment angles.

[0008] Existing methods cannot reasonably handle extreme outliers (Excessive Outliers, EOs) in activation. In the autoregressive Transformer structure of MAR, certain linear layers will produce strongly skewed activation distribution in the step-by-step prediction process, that is, there are extreme activation values (EOs) in a small number of channels that are much higher than the rest of the channels. Existing uniform quantizers have difficulty in making the best choice between "preserving the accuracy of outliers" and "maintaining the accuracy of normal values", resulting in a serious loss of normal information accuracy, eventually accumulating errors and affecting the generated results. Moreover, existing methods cannot deal with amplified anomalies (Amplified Anomalies, AAs) in the diffusion process. In the diffusion network, a small amount of quantization error will be quickly amplified by the amplification factor in the diffusion formula, causing a large drift in the activation amplitude, and eventually forming non-stationary abnormal activation (AAs). This phenomenon is particularly prominent in MAR models, as each step depends on the previous token condition, and the activation distribution changes more dramatically. To this end, the invention specially designs two targeted modules: a hierarchical cluster decoupling mechanism (Hierarchical Cluster Decoupling, HCD) and a scaling recalibration mechanism (Scaling Recalibration, SR) to significantly improve quantization robustness.

[0009] At the same time, existing post-training quantization methods are difficult to achieve stable quantization deployment in mask autoregressive generation models (MAR), mainly affected by factors such as complex model structure and error transmission sensitivity, resulting in a sharp drop in model performance after quantization or even collapse. To this end, the invention first proposes a systematic quantization strategy, which realizes the stable inference deployment of the MAR model under low bit precision (W8A8), significantly reduces the storage and computing costs while maintaining the generation quality, and meets the practical application needs of edge devices and resource-constrained environments.

[0010] The invention proposes a post-training quantization method that stably supports the deployment of mask autoregressive generation models under low bit (such as W8A8) precision. Under the premise of keeping the original model structure and parameters unchanged, two core modules are proposed: a hierarchical cluster decoupling mechanism (Hierarchical Cluster Decoupling, HCD) for identifying and decoupling abnormal activation channels, reducing the impact of EOs; and a scaling recalibration mechanism (Scaling Recalibration, SR) for dynamically adjusting the quantization range of abnormal activation, effectively suppressing the amplification effect of AAs. These two modules act on the Transformer end and the Denoising MLP end, respectively, and cooperate with the static quantization strategy throughout the process to realize the end-side deployment capability without fine-tuning.

[0011] Hierarchical Cluster Decoupling Module (HCD): To address the extremely long-tailed distribution of activations in Transformer stages, the HCD module identifies the "abnormal channels" in the activation space through hierarchical clustering analysis and decouples them from the normal channels. Traditional uniform quantization needs to expand the entire quantization range when facing extreme outliers, resulting in a significant loss of accuracy for the main distribution (main tokens). HCD can achieve independent modeling of abnormal information and accuracy protection. HCD consists of three stages: The first stage (initial clustering): Let the input activation be , representing the activation values of samples on channels. First, extract the maximum activation value of each channel in all samples to form a channel-level feature vector . This vector reflects the strongest response of each channel during inference, which helps identify potential activation abnormal channels. Then, the invention uses a K-Means clustering method optimized by a genetic algorithm to cluster the feature vector, forming a preliminary clustering structure .

[0012] The second stage (main cluster identification): To select the most reasonable number of clusters , the invention defines the main cluster vector set , the non-main cluster vector set , and the within-cluster sum of squares (SSE) as follows:

[0013]

[0014] where is the center of the th cluster. Further considering the quantization effect, the invention introduces a joint optimization objective function as follows, and selects the optimal cluster number :

[0015] The third stage (channel decoupling): According to the clustering results obtained by the optimal cluster number , the channels are divided into main clusters (MainClusters) and non-main clusters (Non-main Clusters), where the main clusters represent normally distributed channels, and the non-main clusters are considered as channels with abnormal activation values. Finally, the main clusters and non-main clusters use different quantization strategies, the main clusters use regular linear quantization, and the non-main clusters can use higher precision or adaptive scaling methods, thereby effectively reducing the overall quantization error caused by abnormal activations.

[0016] Scaling and re-calibration module (SR): The SR module is used to solve the problem that the activation error of the diffusion stage is abnormally amplified. Since in the back diffusion, the activation error of each step will be passed to the next step through the amplification coefficient The slight deviation will cause serious distribution drift. Especially in the early steps, the amplification effect is most significant, causing the output image structure to be seriously distorted. The basic idea of the SR mechanism is as follows: For the diffusion activation We define an abnormal channel set for channels exceeding the Gaussian limit:

[0017] If there is no abnormal channel, the standard quantization operation is directly used. Otherwise, if there is an abnormal channel , the corresponding scaling factor is calculated and the corresponding channel is scaled and quantized after scaling. The calculation method of the scaling factor is:

[0018]

[0019] After quantization, the activation is restored to the original distribution range:

[0020] Compared with the Vanilla quantization strategy which truncates the tail activation, SR retains the tail structure of the activation after rescaling, significantly reduces image artifacts, and improves quantization stability.

[0021] Through the above technical solutions, the present application has the following advantages: (1) The hierarchical clustering decoupling mechanism (HCD) is proposed to identify and decouple the abnormal activation channel, reduce the influence of EOs, and the scaling and recalibration mechanism (SR) is proposed to dynamically adjust the quantization range of abnormal activation, effectively suppressing the amplification effect of AAs.

[0022] (2) HCD and SR act on the Transformer end and the Denoising MLP end respectively, cooperate with the static quantization strategy of the whole process, and realize the end-side deployment ability without fine-tuning.

[0023] (3) A systematic quantization strategy is proposed to realize stable inference deployment of the MAR model under low bit precision (W8A8), significantly reduce the storage and computing cost while maintaining the generation quality, and meet the actual application needs of edge devices and resource-constrained environments. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 MAR model overall quantization process and challenge schematic diagram.

[0025] Figure 2 The application is a post-training quantization method (Q-MAR) flowchart for a mask autoregressive model.

[0026] Figure 3 The hierarchical clustering decoupling mechanism flowchart.

[0027] Figure 4 The principle diagram of the scaling recalibration mechanism. DETAILED DESCRIPTION

[0028] The application will be described in more detail below with examples combined with the accompanying drawings.

[0029] The application is a post-training quantization method suitable for a mask autoregressive generation model, which can stably support the deployment of the mask autoregressive generation model at low bit (such as W8A8) precision. The method systematically designs a solution mechanism for the key problem that the current PTQ fails to apply to complex generative models, as shown in Figure 1 In the existing PTQ method, due to the highly iterative structure of the MAR model using autoregressive generation and diffusion decoding, the activation distribution presents extreme outliers (Excessive Outliers, EOs), and in the reverse diffusion stage, a small quantization error will be amplified by multiple times (Amplified Anomalies, AAs), ultimately causing the model to collapse and the image generation to fail. Therefore, under the premise of keeping the original model structure and parameters unchanged, the application proposes two core modules: a hierarchical clustering decoupling mechanism (Hierarchical Cluster Decoupling, HCD) for identifying and decoupling abnormal activation channels to slow down the influence of EOs; and a scaling recalibration mechanism (Scaling Recalibration, SR) for dynamically adjusting the quantization range of abnormal activation to effectively suppress the amplification effect of AAs. The two modules act on the Transformer end and the Denoising MLP end respectively, and cooperate with the static quantization strategy of the whole process to realize the end-side deployment capability without fine-tuning.

[0030] The application is a post-training quantization method suitable for a mask autoregressive generation model, which mainly includes a hierarchical clustering decoupling module HCD and a scaling recalibration module SR. Please refer to Figure 2 The application is a post-training quantization method (Q-MAR) flowchart for a mask autoregressive model. As can be seen, the method mainly includes 7 steps: S1: Model input stage. The input is a trained mask autoregressive model (MAR) and synthetic label data (such as mask labels, known pixel positions, etc.) for quantization calibration, so as to recover the inference accuracy without relying on additional training.

[0031] S2: Activation statistics extraction phase. In the autoregressive Transformer module of the MAR model, the maximum activation value of each channel is extracted to form a channel feature vector , and hierarchical abnormal channel detection is prepared.

[0032] S3: Abnormal channel identification and decoupling phase. This phase introduces a hierarchical clustering decoupling mechanism, using a GA-optimized K-Means clustering method based on the feature vector to cluster all channels into multiple clusters, and select the optimal cluster number based on clustering error and quantization error . According to the clustering structure, the non-main cluster (i.e. abnormal channel) set is automatically identified , the normal channels and abnormal channels are separated, and their quantization parameters are recorded respectively.

[0033] S4: Diffusion boundary layer anomaly detection phase. On each diffusion step , perform channel-level distribution statistics to determine whether each channel exceeds the Gaussian Bound, forming an abnormal channel set to identify channels with amplification anomalies.

[0034] S5: Scaling and recalibration phase. For activation distributions with abnormal amplification, perform scaling and recalibration mechanism (SR). Normalize and scale all activation values in abnormal channels (subtract mean and then divide by standard deviation), and restore to original scale after quantization to avoid image quality degradation due to activation value truncation.

[0035] S6: Joint quantization processing phase. Normal channels, abnormal channels, and abnormal scaling channels are quantized separately using their respective quantization parameters and quantization schemes to ensure that each type of channel obtains a customized low-error quantization strategy.

[0036] S7: Output quantized model phase. The quantized model after the above processing is output for subsequent deployment or evaluation. The entire process does not involve backpropagation or weight updating, and has high universality and deployment friendliness.

[0037] Specifically, the hierarchical clustering decoupling module addresses the extreme long-tail problem of activation distribution in the Transformer phase by hierarchical clustering analysis, identifies "abnormal channels" in the activation space, and decouples them from the regular channels. Traditional uniform quantization needs to expand the entire quantization range when facing extreme outliers, resulting in a significant loss of accuracy in the main distribution (main token). HCD can achieve independent modeling and accuracy protection of abnormal information. Please refer to Figure 3 , HCD consists of three phases: The first stage (initial clustering): Let the input activations be , representing the activation values of samples on channels. First, we extract the maximum activation value of each channel in all samples to form a channel-level feature vector . This vector reflects the strongest response of each channel in the inference process, which helps to identify potential abnormal activation channels. Subsequently, we use the K-Means clustering method optimized by a genetic algorithm to cluster the feature vector, preliminarily forming a clustering structure .

[0038] The second stage (main cluster discrimination): To select the most reasonable number of clusters , we define the main cluster vector set , the non-main cluster vector set , and the within-cluster sum of squares (SSE) as follows:

[0039]

[0040] where is the center of the th cluster. Further considering the quantization effect, we introduce a joint optimization objective function as follows to select the optimal number of clusters :

[0041] The third stage (channel decoupling): According to the clustering results obtained from the optimal number of clusters , the channels are divided into main clusters (MainClusters) and non-main clusters (Non-main Clusters), where the main clusters represent normally distributed channels, and the non-main clusters are considered as channels with abnormal activation values. Finally, different quantization strategies are used for main cluster and non-main cluster channels, respectively. The main clusters use regular linear quantization, while the non-main clusters can use higher precision or adaptive scaling methods, thereby effectively reducing the overall quantization error caused by abnormal activation.

[0042] Specifically, the scaling and recalibration module is used to solve the problem of abnormal amplification of activation error in the diffusion stage. Since in the backward diffusion, the activation error of each step is transmitted to the next step through the amplification coefficient , a small deviation can cause serious distribution drift. Especially in the early steps, the amplification effect is most significant, leading to serious structural distortion of the output image. Please refer to Figure 4 , the basic idea of the SR mechanism is as follows: For the diffusion activation , we define an abnormal channel set For channels beyond the Gaussian bound:

[0043] If there is no abnormal channel, directly use the standard quantization operation. Otherwise, if there is an abnormal channel , calculate the corresponding scaling factor and then scale and quantize the corresponding channel. The calculation method of the scaling factor is:

[0044]

[0045] After quantization, the activation is restored to the original distribution range:

[0046] Compared with the Vanilla quantization strategy which truncates the tail of the activation, SR retains the tail structure of the activation after rescaling, significantly reduces image artifacts, and improves quantization stability.

[0047] In order to illustrate the technical advancement of the method of the present application, the method of the present application is compared with the traditional method. Using multiple classical baseline methods which are very effective in previous models, under two different quantization settings of the MAR model, using the parameter combination of 64 and 32 autoregressive iteration times respectively and keeping the denoising time step unchanged at 100 steps, experiments are carried out to verify the effectiveness and robustness of the method.

[0048] Among them, the symbol o represents that the boundary layer of the Diffusion module in the MAR model does not perform quantization processing. This actually takes into account that the quantization error caused by quantization of the boundary layer is too large, so that all baseline methods are completely ineffective and cannot reflect the initial effect of the baseline method. From these results, it can be seen that our method Q-MAR still has more prominent model performance compared with these baseline methods, and the generation ability far exceeds these classical methods. The symbol represents that all linear layers and matrix multiplication calculations in the MAR model are quantized to obtain a comprehensive and comprehensive comparison result. From these comparisons, it can be seen that the generation ability of our method Q-MAR after quantization greatly improves the problem of model performance collapse caused by model quantization, and still has a high level of generation ability. The detailed comparison of the performance results of the iteration times of 64 and 32 is shown in Table 1 and Table 2.

[0049] Table 1 Performance with 64 iterations and 100 comparison time steps

[0050] Table 2 Performance comparison with 32 iterations and 100 time steps

Claims

1. A post-training quantization method suitable for an autoregressive mask generation model, characterized in that, Comprise the following steps: S1: model input, input is a trained mask autoregressive model (MAR), and mask label for quantization calibration, known pixel position label, so as to restore inference accuracy without relying on additional training; S2: activate statistics extraction, in the autoregressive Transformer module of the MAR model, set the input activation as , represents the activation value of samples on channels, extract the maximum activation value of each channel to form a channel feature vector , and prepare for hierarchical abnormal channel detection; S3: Abnormal channel identification and decoupling, introduce hierarchical clustering decoupling mechanism (HCD), use genetic algorithm optimized K-Means clustering method, based on feature vector All channels are clustered into multiple clusters, and the optimal cluster number is selected according to the clustering error and quantization error , automatically identify the non-main cluster (i.e. abnormal channel) set according to the clustering structure , realize the separation of normal channel and abnormal channel, and record their quantization parameters respectively; S4: Diffusion boundary layer anomaly detection, at each diffusion activation Channel level distribution statistics are performed on the above to determine whether each channel exceeds the Gaussian Bound , forming an anomaly channel set to identify channels that have amplification anomalies S5: scaling re calibration, for the existence of abnormal amplification of activation distribution, scaling re calibration mechanism (SR) is carried out, all activation values in abnormal channel are normalized and scaled, and are restored to the original scale after quantization, so as to avoid image quality degradation caused by the truncation of activation values; S6: joint quantization processing, respectively using the respective quantization parameters and quantization schemes of normal channel, abnormal channel and abnormal scaling channel to carry out post training quantization, so as to ensure that each type of channel obtains customized low error quantization strategy; S7: output the quantized model, output the quantized model processed above Output, for subsequent deployment or evaluation, the entire process does not involve back propagation or weight update, with high universality and deployment friendliness.

2. The post-training quantization method for autoregressive mask generation model according to claim 1, wherein, The step S4 of collecting the abnormal channel set The calculation method is as follows: 。 3. The post-training quantization method for autoregressive mask generation model according to claim 1, wherein, The normalization scaling processing in the step S5 refers to subtracting mean value and dividing by standard deviation, and the calculation method of scaling factor is: ， Wherein, 。

Images