Abnormal value-based variable granularity ViTs model post-training quantification method, electronic equipment and storage medium

Abstract

The invention discloses a post-training quantization method of a variable granularity ViTs model based on an abnormal value, electronic equipment and a storage medium, which can reduce quantization loss and improve model performance. The method comprises the following steps: 1, converting a floating-point number into a low-order fixed-point number in a quantization process; 2, a # imgabs 0 # Transformer block serves as a reconstruction unit, calculation of a quantization error is defined as # imgabs 1 #, # imgabs 2 # and # imgabs 3 # represent input activation and quantization of the # imgabs 4 # Transformer block respectively, # imgabs 5 # and # imgabs 6 # represent weights and quantization of the # imgabs 7 # Transformer block respectively, # imgabs 8 # represents expectation, # imgabs 9 # represents a Frobenius norm, and # imgabs 10 # represents a quantization function of the # imgabs 11 # Transformer block; and 3, calculating gradient estimation of rounding operation by adopting a straight-through estimator during back propagation.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network compression and acceleration, and in particular to a post-training quantization method, electronic device and storage medium of a variable granularity ViTs model based on outliers. Background Art

[0002] With the rapid development of deep learning technology, the application of Transformer architecture in visual tasks has become more and more widespread. This architecture stacks modules layer by layer, so that the performance of the model continues to improve as the model size increases, but it also brings huge demands on computing resources. Therefore, researchers began to explore how to reduce the size of the Transformer model and accelerate the reasoning process through various model compression techniques. Common methods include neural architecture search, model pruning, and model quantization.

[0003] Quantization technology is a process of converting the weights and activation values ​​of a model from floating-point numbers to low-bit fixed-point numbers, which can significantly reduce the size of the model and improve its access efficiency. Especially on hardware that supports low-bit operations, quantized models can significantly improve inference speed. Quantization-aware training (QAT) and post-training quantization (PTQ) are the two main quantization methods, and they have different costs when implementing quantized models. The latter is particularly suitable for large-scale models due to its low data dependence and low algorithmic overhead, but its efficiency often decreases with the improvement of model performance, especially when dealing with data outliers in ViTs, uniform quantization does not work well. In response to these problems, recent studies have proposed a variety of improvement solutions, such as: (1) FQ-ViT solves the problem of poor distribution of attention maps by using a log2 quantizer and introduces an exponential function integer approximation method for softmax quantization; in addition, FQViT combines the quadratic quantization of the layer norm layer to achieve pure integer reasoning of ViTs.

[0004] (2) PTQ4ViT introduces double uniform quantization after softmax and activation functions to handle outlier distribution.

[0005] (3) APQ-ViT optimizes performance by prioritizing the losses caused by outliers during the calibration process.

[0006] (4) RepQ-ViT proposes log√2 quantization to better adapt to the distribution of softmax output activations, and transitions from log√2 quantization and per-channel activation quantization to log2 quantization and per-layer activation quantization through reparameterization technology, thereby improving inference efficiency.

[0007] (5) Quantible Transformers uses a trimmed softmax to prevent gradient accumulation in pre-training, and selectively updates the representation of specific tags through gated attention, so that there are no outliers in the pre-trained model, which is more suitable for quantization.

[0008] However, most current methods ignore the importance of granular reconstruction during ViTs quantization. MRECG explores the role of granular reconstruction in reducing loss fluctuations in convolutional neural networks, but this assumption of topological homogeneity does not apply to different modules within and between Transformer blocks. Therefore, it is necessary to deeply study the characteristics of ViTs and explore the impact of these characteristics on quantization performance. Summary of the invention

[0009] The purpose of the present invention is to provide a post-training quantization method, electronic device and storage medium of a variable granularity ViTs model based on outliers, which effectively solves the problems caused by outliers, helps to reduce quantization losses and improve model performance.

[0010] In order to achieve the above object, the solution of the present invention is: A post-training quantization method for a variable granularity ViTs model based on outliers, comprising the following steps: Step 1: In the process of quantizing the ViTs model, the floating point number is converted into a low-order fixed point number. The quantization function is as follows: ; in, is the quantization function The input, i.e. the parameters that need to be quantified; For truncation operation, For rounding operations, , and Represent step size, zero point and bit width respectively; Step 2: The Transformer block is used as the reconstruction unit, and the calculation method of the quantization error is defined as follows: ; in, and Respectively represent Input activations and their quantization for each Transformer block; and Respectively represent The weights of each Transformer block and their quantization; express expectations; represents the Frobenius norm; Indicates Quantization function of a Transformer block; Step 3: Use the straight-through estimator to calculate the gradient estimate of the rounding operation during the back-propagation process.

[0011] An electronic device comprises a processor, a memory and an application; the application is stored in the memory and is configured to execute the post-training quantization method of the variable granularity ViTs model based on outliers as claimed in claim 1 by the processor.

[0012] A computer-readable storage medium storing a computer program; when the computer program is executed in the computer, the computer executes the post-training quantization method of the variable granularity ViTs model based on outliers as claimed in claim 1.

[0013] After adopting the above technical solution, the present invention has the following technical effects: The present invention can realize the "granularity reconstruction" of the ViTs model and is specifically used for the post-training quantization of the ViTs model. Through more detailed optimization steps, it can effectively solve the problems caused by outliers to significantly improve the accuracy of the model; by increasing the number of granularity combinations (i.e., the number of reconstructed Transformer blocks), it helps to reduce quantization loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of granularity segmentation according to a specific embodiment of the present invention; Figure 2 This is the comparison data of the Top-1 accuracy of the present invention and other methods on ImageNet. DETAILED DESCRIPTION

[0015] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0016] Typically, the quantization function of the model The specific definition is as follows: ; in, is the quantization function Input; For truncation operation, For rounding operations, , and Represent the step size, zero point and bit width respectively. It can be concluded from the quantization function that when the input When the distribution is uniform, the theoretical minimum quantization error can be achieved.

[0017] The quantization error can usually be decomposed into two types: one is the truncation error caused by the value exceeding the quantization range, and the other is the rounding error caused by numerical rounding.

[0018] In the ViTs model, it is found that there are outlier problems in the activation values, and these outliers tend to become more significant in subsequent Transformer blocks. Since the distribution of fixed points after quantization is uniform, when these outliers are not truncated by the maximum quantization fixed point, the activation values ​​of other channels will be mapped to zero in large quantities, resulting in a large number of rounding errors. Outliers bring huge challenges to the post-training quantization process of the ViTs model. At the same time, without considering outliers, the quantization error is mainly determined by the truncation error, so only a small number of activation values ​​will be mapped to uniform quantization fixed points distributed between outliers and normal values. The existence of outliers "compresses" the representation range of the remaining normal values, thereby increasing the difficulty of quantization.

[0019] Although some reconstruction methods have been used for post-training quantization of convolutional neural networks, these methods mainly guide the intermediate agent through the objective function to minimize the quantization error, which is not necessarily applicable to the ViTs model.

[0020] refer to Figure 1 As shown, the present invention discloses a post-training quantization method of a variable granularity ViTs model based on outliers, comprising the following steps: Step 1: In the process of quantizing the ViTs model, the floating point numbers are converted to low-order fixed point numbers to reduce the model size and enhance the efficiency of data access. The quantization function is as follows: ; in, is the quantization function The input, i.e. the parameter that needs to be quantified; For truncation operation, For rounding operations, , and Represent step size, zero point and bit width respectively; Step 2: The Transformer block is used as the reconstruction unit, and the calculation method of the quantization error is defined as follows: ; in, and Respectively represent Input activations and their quantization for each Transformer block; and Respectively represent The weights of each Transformer block and their quantization; express expectations; represents the Frobenius norm, which is used to measure the quantization error; Indicates Quantization function of a Transformer block; Step 3: In order to deal with the non-differentiability caused by rounding operations, the straight-through estimator (STE) is used to calculate the gradient estimate of the rounding operation during the back-propagation process, thereby effectively dealing with the non-differentiability problem. This allows the quantization error to reach the minimum value under uniformly distributed input conditions, thereby optimizing the quantization performance of the model.

[0021] Through the above scheme, the present invention can realize the "granular reconstruction" of the ViTs model, which is specifically used for the post-training quantization of the ViTs model. Through more detailed optimization steps, the problems caused by outliers can be effectively solved to significantly improve the accuracy of the model; by increasing the number of granularity combinations (that is, the number of reconstructed Transformer blocks), it is helpful to reduce quantization loss.

[0022] In order to verify the technical effect of the present invention, the following experiments were performed: (1) During the experiment, the candidate granularity reconstructions of all models were restricted to combinations of three Transformer blocks, and a large number of granularity reconstruction schemes were generated through random sampling.

[0023] (2) Subsequently, the quantization accuracy is restored through an optimization process; the results show that the number of combinations is negatively correlated with the final block loss of all ViTs. This indicates that coarser granularity reconstruction can effectively reduce the quantization loss of the model.

[0024] See also Figure 2 ,The experimental results show that the use of coarser granularity reconstruction can ,effectively solve the outlier problem in ViTs. On the contrary, the Swin model ,achieves better quantitative performance when using finer granularity reconstruction, ,confirming our previous findings.

[0025] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0026] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0027] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A post-training quantization method for a variable-granularity ViTs model based on outliers, characterized in that The following steps are involved: Step 1: In the process of quantizing the ViTs model, the floating point number is converted into a low-order fixed point number. The quantization function is as follows: ； in, is the quantization function The input, i.e. the parameter that needs to be quantified; For truncation operation, For rounding operations, , and Represent step size, zero point and bit width respectively; Step 2: The Transformer block is used as the reconstruction unit, and the calculation method of the quantization error is defined as follows: ； in, and Respectively represent Input activations and their quantization for each Transformer block; and Respectively represent The weights of each Transformer block and their quantization; express expectations; represents the Frobenius norm; Indicates Quantization function of a Transformer block; Step 3: Use the straight-through estimator to calculate the gradient estimate of the rounding operation during the back-propagation process.

2. An electronic device, characterized in that The invention comprises a processor, a memory and an application program; the application program is stored in the memory and is configured to be executed by the processor as claimed in claim 1 for executing the post-training quantization method of the variable granularity ViTs model based on outliers.

3. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer is caused to execute the post-training quantization method of the variable granularity ViTs model based on outliers as claimed in claim 1.

Images