Two-stage long tail compensation logarithmic quantization method of ViTs
By using a two-stage long-tail compensation quantization method and a high-efficiency shift quantizer in the ViTs model, the poor quantization performance caused by the long-tail distribution of the activation value is solved, and higher quantization accuracy and applicability are achieved.
Patent Information
- Application Number
- CN202411520186.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-02-18
AI Technical Summary
The existing quantization methods are difficult to effectively solve the long-tail distribution problem of activation values in the ViTs model, resulting in poor quantization performance.
A two-stage long-tail compensation pair quantization method is proposed. The activation values are dynamically divided into long-tail and non-long-tail parts by adaptive operators, and quantization is carried out using a shift-efficient quantizer to achieve a more refined quantitative strategy.
The applicability and accuracy of quantization are significantly improved, especially when processing sparse activation values, and the ability to characterize long-tail distributions is improved. The experimental results show that it is better than the existing baseline model on multiple indicators.
Smart Images
Figure CN119476365A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of visual converter quantization, and in particular to a two-level long-tail compensation logarithmic quantization method of ViTs. Background Art
[0002] Since its introduction, Transformer has quickly become the most prominent model architecture in language models. This new network structure has improved major translation tasks and set many new records. With the introduction of Visual Transformer (ViT), its influence has been significantly expanded. Vision Transformers (ViTs) have demonstrated surprising potential in a range of visual applications such as image classification, object detection, and semantic segmentation, illustrating the effectiveness and versatility of Transformer-based architectures. Compared with convolutional neural networks (CNNs) across different visual tasks, Transformer architectures have demonstrated their ability to achieve competitive performance benchmarks and have therefore been widely studied as new visual backbone networks. Due to its powerful performance, Transformer has become the mainstream architecture for deep learning. To date, several high-performance generative AI models, such as OpenAI's GPT-4 and Google's Bard, are based on Transformer architecture models. As deep learning has developed to this day, Transformer-based LLM models have gradually shown their shortcomings in terms of reasoning efficiency, parallel computing capabilities, and long sequence modeling capabilities. Faced with massive amounts of computing data, complex downstream tasks, and higher performance requirements, how to achieve faster training, lower inference overhead, and stronger model performance is one of the "core issues" in the current field of artificial intelligence research.
[0003] Compared with CNN, the fully connected layers and self-attention layers in the Transformer architecture rely on a lot of intensive calculations and have more parameters, resulting in unbearable memory usage, power consumption, and inference latency. Faced with the challenge of directly deploying the model will incur huge resource costs, many compression and acceleration methods have been applied to the Transformer architecture. Among them, model compression is an effective strategy for reducing the development cost of the Transformer model, including pruning, quantization, knowledge distillation, and efficient architecture design. Quantization reduces development costs by representing model weights and intermediate features with lower bits. Model quantization technology has become an important research direction.
[0004] Despite good progress in designing better PTQ quantizers for convolutional neural networks (CNNs), the performance of these schemes for quantizing ViTs models using this approach, especially activation quantization, is far from satisfactory. Transformer layers generate up to millions of activation values with complex distributions. For example, the output of the GELU function has an asymmetric distribution, with spikes at certain values in the histogram and a long tail over a large range. Certain linear projection layers also result in very large activation values that are sparsely distributed with a very long tail. Figure 1 As shown in the figure, it can be observed that whether it is the more common linear layer in the model or the unique architecture in ViT, the activation values show a long-tail distribution, and the long-tail characteristics are that most values are concentrated near zero, and the long-tail part is far away from zero. At present, the existing methods have not found a general solution to the long-tail distribution problem of activation values in ViTs. Summary of the invention
[0005] The purpose of the present invention is to provide a long-tail enhanced logarithmic quantization TelQ-ViT (Tail-EnhancedLogarithmic Quantization) method suitable for the ViTs architecture, in view of the fact that the existing methods cannot solve the long-tail distribution problem of activation values in ViTs. The two-step non-uniform quantization method tailored for ViTs can achieve more accurate and more general post-training quantization for ViTs.
[0006] The technical solution of the present invention is as follows:
[0007] A two-level long-tail compensation logarithmic quantization method for ViTs, comprising the following steps:
[0008] Adaptive compensation long-tail enhancement uses different processing strategies for different parts of activation values through adaptive operators. According to the adaptive operator obtained during calibration, the activation values are dynamically divided into two parts: one part is the value close to zero, which maintains a high quantization accuracy during the first logarithmic quantization process; the other part is distributed in the long-tail area and is quantized through the second adaptive compensation.
[0009] Shifting efficient long tail enhancement, through shifting efficient Quantizer,reparameterizes the scaling factor, transforms the logarithm base to 2, and completes efficient shift operations during inference.
[0010] Furthermore, the adaptive compensation long tail enhancement includes the following steps:
[0011] The data is divided into two parts through the adaptive operator threshold: one part is the activation value whose absolute value of the quantization error is greater than the threshold, and secondary quantization is used to compensate its quantization accuracy; the other part is the complement of the first part, that is, the activation value whose absolute value of the quantization error is less than the threshold. The quantization error of this part is small and does not need secondary quantization compensation.
[0012] Furthermore, the specific steps of the adaptive compensation long-tail enhancement include:
[0013] The activation value X is first obtained by the first log quantization Activation value passes The quantization process obtains an integer quantized value;
[0014] Then through sign(q)S·2 -|q| The dequantization process obtains the floating point quantized value, which is then passed through X and The difference between the two is the error E of the first quantization;
[0015] The adaptive operator threshold obtained by the hyperparameter R will be It is divided into two parts, in which the quantization error greater than the threshold is quantized for the second compensation, and the error value less than the threshold is ignored, and the error after quantization compensation is obtained. With the value after a quantization Add them together to get the final quantized result.
[0016] Furthermore, the quantization formula of the activation value X is:
[0017] E = X - log Q (X),
[0018] threshold=percentile(|E|,R%),
[0019]
[0020] Where E is a tensor of the same shape as the activation value X, R is a hyperparameter, threshold is a scalar greater than 0, representing the R-th percentile value in |E|. For each element x in X, if the quantization error of the first quantization is greater than the threshold, the quantization difference needs to be requantized, otherwise, no requantization is required.
[0021] Furthermore, the adaptively compensated long-tail enhancement uses a log2 quantizer for non-uniform quantization, and the log2 quantizer includes:
[0022] The N-bit integer q is the logarithm of the real number x in [-1,1], and the range of q is [-M,M-1], where M = 2 N-1 , the quantized value of x It is expressed as:
[0023]
[0024] The logarithmic quantization process is expressed as:
[0025]
[0026] sign() is the sign function, s∈R + is the scaling factor, clip(x,a,b) is the clipping function, which clips the value of x to the interval [a,b]. It can be expressed as
[0027] logQ(x):Dequant(Quant(x)).
[0028] Furthermore, the shift-efficient long-tail enhancement is accomplished by reparameterizing the scaling factor The conversion from quantizer to log 2 quantizer includes the basis transformation method of the quantization process and the basis transformation method of the dequantization process.
[0029] Furthermore, the basis transformation method of the quantization process includes:
[0030] Given activation value A and The scaling factor s∈R of the quantizer 1 , in the formula, only the case where the activation value is non-negative is considered. According to the basis change formula of the logarithmic function, the quantization process is:
[0031]
[0032] Therefore, for the quantization process, the conversion to a log2 quantizer can be done by simply multiplying by a constant factor.
[0033] Furthermore, the basis transformation method of the dequantization process includes:
[0034] Using the basis transformation formula of the pow function, we get a form based on 2. The four cases are calculated separately:
[0035]
[0036]
[0037] in Represents the top function, which can ensure is always an integer. V(·) is the parity indicator function, which returns 0 for even numbers and 1 for odd numbers. The parity indicator function and its coefficients can be merged into s to obtain the reparameterized scaling factor. as follows:
[0038]
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] 1. A two-level long-tail compensation logarithmic quantization method for ViTs. Aiming at the long-tail sparse distribution of ViTs activation values, a general non-uniform quantization method is proposed, which is applicable to all components in the Transformer block, effectively improving the applicability and accuracy of quantization;
[0041] 2. A two-level long-tail compensation logarithmic quantization method for ViTs. Aiming at the non-uniform parameter distribution characteristics in ViTs, a novel non-uniform post-training quantization (PTQ) framework is proposed. This framework introduces an adaptive compensation mechanism to dynamically divide the activation values into long-tail and non-long-tail parts during the quantization process, thereby achieving a more refined quantization strategy. For the densely distributed non-long-tail part, this method retains a high quantization accuracy; and for the small but important long-tail part, the adaptive compensation quantization greatly improves the ability to characterize sparse activation values;
[0042] 3. A two-level long-tail compensation logarithmic quantization method for ViTs, which proposes a shift-efficient The quantizer performs corresponding shift and re-parameter operations according to different quantization results by designing stage functions, so that it can maintain the original data distribution in the case of 6-bit quantization while achieving efficient reasoning; TelQ-ViT is evaluated on image classification tasks using a variety of ViT architectures. Experimental results show that it outperforms existing baseline models in multiple indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is a partial activation value distribution diagram in ViTs.
[0044] Figure 2 This is the overall architecture diagram of the adaptive compensation long-tail enhancement method of this application.
[0045] Figure 3 The experimental results of this application method Figure 1 .
[0046] Figure 4 The experimental results of this application method Figure 2 . DETAILED DESCRIPTION
[0047] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0048] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.
[0049] See also Figure 1-4 , a two-level long-tail compensation logarithmic quantization method for ViTs, comprising the following steps:
[0050] Adaptive compensation long-tail enhancement uses different processing strategies for different parts of activation values through adaptive operators. According to the adaptive operator obtained during calibration, the activation values are dynamically divided into two parts: one part is the value close to zero, which maintains a high quantization accuracy during the first logarithmic quantization process; the other part is distributed in the long-tail area and is quantized through the second adaptive compensation.
[0051] The adaptively compensated long-tail enhancement uses a log2 quantizer for non-uniform quantization. The log2 quantizer includes:
[0052] The N-bit integer q is the logarithm of the real number x in [-1,1], and the range of q is [-M,M-1], where M = 2 N-1 , the quantized value of x It is expressed as:
[0053]
[0054] The logarithmic quantization process is expressed as:
[0055]
[0056] sign() is the sign function, s∈R + is the scaling factor, clip(x,a,b) is the clipping function, which clips the value of x to the interval [a,b]. It can be expressed as
[0057] logx(x):Dequant(Quant(x)).
[0058] The adaptive compensation long-tail enhancement includes the following steps:
[0059] The data is divided into two parts through the adaptive operator threshold: one part is the activation value whose absolute value of the quantization error is greater than the threshold, and secondary quantization is used to compensate its quantization accuracy; the other part is the complement of the first part, that is, the activation value whose absolute value of the quantization error is less than the threshold. The quantization error of this part is small and does not need secondary quantization compensation.
[0060] The overall architecture of adaptive compensation long tail enhancement is shown in the figure below. Figure 2 As shown. The activation value X is first obtained by the first log quantization Activation value passes The quantization process obtains an integer quantized value;
[0061] Then through sign(q)S·2 -|q| The dequantization process obtains the floating point quantized value, which is then passed through X and The difference between the two is the error E of the first quantization;
[0062] The adaptive operator threshold obtained by the hyperparameter R will be It is divided into two parts, in which the quantization error greater than the threshold is quantized for the second compensation, and the error value less than the threshold is ignored, and the error after quantization compensation is obtained. With the value after a quantization Add them together to get the final quantized result.
[0063] The quantization formula of the activation value X is:
[0064] E = X - log Q (X),
[0065] threshold=percentile(|E|,R%),
[0066]
[0067] Where E is a tensor of the same shape as the activation value X, R is a hyperparameter, threshold is a scalar greater than 0, representing the R-th percentile value in |E|. For each element x in X, if the quantization error of the first quantization is greater than the threshold, the quantization difference needs to be requantized, otherwise, no requantization is required.
[0068] Since the quantization error of most activation values is small after the first quantization, in practical applications, only a small number of activation values need to be quantized and compensated. Although the secondary compensation process adds some computational overhead, due to the characteristics of the log2 quantizer, a large number of multiplication operations in the inference process can be converted into efficient shift operations. Therefore, compared with the traditional one-time uniform quantization, the adaptive compensation long-tail enhancement only slightly increases the overall computational complexity. Considering the significant performance improvement it brings, this additional overhead is acceptable. For example, in the 4-bit quantization of ViT-S, when only the log quantizer is used, the Top-1 accuracy is 61.64%, while when the proposed adaptive compensation long-tail enhancement method is applied and R is set to 95, that is, the activation values with the absolute value of the quantization error greater than the 95th percentile of the overall quantization error are compensated for secondary quantization, the Top-1 accuracy is increased to 75.51%, an increase of about 13.87%. This improvement shows the significant improvement of the method in quantization accuracy.
[0069] Shifting efficient long tail enhancement, through shifting efficient Quantizer,reparameterizes the scaling factor, transforms the logarithm base to 2, and completes efficient shift operations during inference.
[0070] Shift-efficient long-tail enhancement is accomplished by reparameterizing the scaling factor The conversion from quantizer to log 2 quantizer includes the basis transformation method of the quantization process and the basis transformation method of the dequantization process.
[0071] The basis transformation methods of the quantization process include:
[0072] Given activation value A and The scaling factor s∈R of the quantizer 1 , in the formula, only the case where the activation value is non-negative is considered. According to the basis change formula of the logarithmic function, the quantization process is:
[0073]
[0074] Therefore, for the quantization process, the conversion to a log2 quantizer can be done by simply multiplying by a constant factor.
[0075] The basis transformation methods for the dequantization process include:
[0076] Using the basis transformation formula of the pow function, we get a form with base 2, but the new exponent term -A^Q / 4 cannot be guaranteed to be an integer required to perform the shift operation. Therefore, for the dequantization process, The four cases are calculated separately:
[0077]
[0078] in Represents the top function, which can ensure is always an integer. V(·) is the parity indicator function, which returns 0 for even numbers and 1 for odd numbers. The parity indicator function and its coefficients can be merged into s to obtain the reparameterized scaling factor. as follows:
[0079]
[0080] Finally, due to The dequantization process can also benefit from the efficient shift operation. It is worth noting that compared with the previous scaling factor s, the reparameterized scaling factor Only slight additional computational overhead is introduced during inference, since the parity indicator function and shift operations can be performed by efficient circuits.
[0081] experiment:
[0082] Experiments were conducted under 4-bit quantization and 6-bit quantization. The dataset was imagenet, and the value of R was set to 95%. A two-stage compensated logarithmic quantization method was applied to the activation values except the embedding layer. The settings under 4-bit quantization were: the base of the first log quantization was 2, and the base of the quantization compensation was The setting for 6-bit quantization is that both quantizations use shift-efficient Quantizer. All weights are uniformly quantized channel by channel, and only 32 random images are used to calibrate the quantization parameters. The following experimental results are obtained.
[0083] like Figure 3 and Figure 4 shown.
[0084]
[0085]
[0086]
[0087] TelQ-ViT is evaluated on image classification tasks using various ViT architectures, and experimental results show that it outperforms existing baseline models in multiple metrics.
[0088] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.
Claims
1. A two-level long-tail compensation logarithmic quantization method for ViTs, characterized in that: The following steps are involved: Adaptive compensation long-tail enhancement uses different processing strategies for different parts of activation values through adaptive operators. According to the adaptive operator obtained during calibration, the activation values are dynamically divided into two parts: one part is the value close to zero, which maintains a high quantization accuracy during the first logarithmic quantization process; the other part is distributed in the long-tail area and is quantized through the second adaptive compensation. Shifting efficient long tail enhancement, through shifting efficient Quantizer,reparameterizes the scaling factor, transforms the logarithm base to 2, and completes efficient shift operations during inference.
2. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 1, characterized in that: The adaptive compensation long tail enhancement comprises the following steps: The data is divided into two parts through the adaptive operator threshold: one part is the activation value whose absolute value of the quantization error is greater than the threshold, and secondary quantization is used to compensate its quantization accuracy; the other part is the complement of the first part, that is, the activation value whose absolute value of the quantization error is less than the threshold. The quantization error of this part is small and does not need secondary quantization compensation.
3. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 2, characterized in that: The specific steps of the adaptive compensation long-tail enhancement include: The activation value X is first obtained by the first log quantization Activation value passes The quantization process obtains an integer quantized value; Then through sign(q)S·2 -|q| The dequantization process obtains the floating point quantized value, which is then passed through X and The difference between the two is the error E of the first quantization; The adaptive operator threshold obtained by the hyperparameter R will be It is divided into two parts, in which the quantization error greater than threshold is quantized for the second time, and the error value less than threshold is ignored, and the error after quantization compensation is obtained. With the value after a quantization Add them together to get the final quantized result.
4. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 3, characterized in that: The quantization formula of the activation value X is: E = X - log Q (X), Where E is a tensor of the same shape as the activation value X, R is a hyperparameter, threshold is a scalar greater than 0, representing the R-th percentile value in |E|. For each element x in X, if the quantization error of the first quantization is greater than the threshold, the quantization difference needs to be requantized, otherwise, no requantization is required.
5. A two-level long-tail compensation logarithmic quantization method for ViTs according to any one of claims 2 to 4, characterized in that: The adaptively compensated long-tail enhancement adopts a log2 quantizer for non-uniform quantization, and the log2 quantizer includes: The N-bit integer q is the logarithm of the real number x in [-1,1], and the range of q is [-M,M-1], where M = 2 N-1 , the quantized value of x It is expressed as: The logarithmic quantization process is expressed as: sign() is the sign function, s∈R + is the scaling factor, clip(x,a,b) is the clipping function, which clips the value of x to the interval [a,b]. It can be expressed as logQ(x):Dequant(Quant(x)).
6. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 1, characterized in that: The shift-efficient long-tail enhancement is accomplished by reparameterizing the scaling factor The conversion from quantizer to log 2 quantizer includes the basis transformation method of the quantization process and the basis transformation method of the dequantization process.
7. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 6, characterized in that: The basis transformation method of the quantization process includes: Given activation value A and The scaling factor s∈R of the quantizer 1 , in the formula, only the case where the activation value is non-negative is considered. According to the basis change formula of the logarithmic function, the quantization process is: Therefore, for the quantization process, the conversion to a log2 quantizer can be done by simply multiplying by a constant factor.
8. A two-level long-tail compensation logarithmic quantization method for ViTs according to claim 6, characterized in that: The basis transformation method of the dequantization process includes: Using the basis transformation formula of the pow function, we get a form based on 2. The four cases are calculated separately: in Represents the top function, which can ensure is always an integer. V(·) is the parity indicator function, which returns 0 for even numbers and 1 for odd numbers. The parity indicator function and its coefficients can be merged into s to obtain the reparameterized scaling factor. as follows: