A quantization method for a vision transformer neural network

Abstract

The application discloses a kind of efficient quantization methods for Vision Transformer neural network, it includes S1, using image classification dataset to train Vision Transformer neural network, obtain full-precision model;S2, randomly select the preset quantity of picture in the training set of image classification dataset input full-precision model, the maximum and minimum of the weight and activation value of each layer of model are counted, as quantization range;S3, according to the quantization range of each layer weight and activation value, respectively to the weight and activation value of corresponding layer is asymmetrically quantized;S4, after the activation value of Softmax is based on the double granularity quantization of hessian matrix, the scale reparameterization quantization based on channel optimization is carried out to layer normalization layer activation value, and the quantized neural network model is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to network quantization technology, and in particular to a quantization method for a Vision Transformer neural network. BACKGROUND

[0002] With the development of deep learning, many new neural network architectures have emerged, such as Transformer, Diffusion, etc. Inspired by the success of Transformer in natural language processing (NLP) tasks, researchers have extended its application to various computer vision (CV) tasks and developed a series of Vision Transformer (ViT) models. Now ViT has achieved advanced performance in image classification, object detection, semantic segmentation, etc. However, ViT relies on heavy and intensive computation, which inevitably leads to a large amount of memory usage, power consumption and inference delay. These high overheads hinder the deployment of ViT on resource-constrained edge devices. Therefore, model compression of ViT is crucial in practical applications, especially in scenarios that require low-cost deployment and real-time inference.

[0003] Model quantization is an effective and popular model compression scheme. Model quantization converts high-precision weights and activation values into low-precision values through some mapping, thereby reducing model complexity and computational complexity, making it more efficient for deployment in resource-constrained environments. On image classification tasks, there are now many related studies on post-training quantization (PTQ) of ViT. They identify two major challenges in quantizing ViT: the distribution of activation values after the Softmax activation function deviates significantly from the Gaussian distribution, and the significant inter-channel variance of activation values after the Layer Normalization layer.

[0004] To address the deviation of the distribution of activation values after the Softmax activation function from the Gaussian distribution, RepQ-ViT uses The quantizer quantizes the activation values after the Softmax activation function. The introduction of the quantizer increases the computational load and delay in the inference process. This increase in computational load makes such methods less suitable for hardware deployment, especially on resource-constrained devices. In addition, due to the presence of a large number of outliers in the activation value distribution, quantization of activation values after the Softmax activation function often leads to large errors.

[0005] For the significant inter-channel variance of the activation value after the layer normalization layer, a very ideal quantization scheme is to perform channel-wise quantization, but the channel quantization hardware is incompatible and is not easy to deploy. RepQ-ViT decouples the quantization and inference stages, and rescales the channel quantization to layer-wise quantization in the inference stage, which can make the layer-wise quantization accuracy of the activation value after the layer normalization layer approximate to the channel-wise quantization, so that it can be better deployed.

[0006] Although the method proposed by RepQ-ViT approximates the channel-wise quantization through layer-wise quantization, it largely solves the inter-channel difference problem, but there is still a certain gap compared with the performance of the channel-wise quantization. Compared with the channel-wise quantization, the scale reparameterization method will cause the classification accuracy to decrease due to the existence of outliers in each channel of the activation value after the layer normalization layer and the reparameterization weight. SUMMARY

[0007] In view of the above problems in the prior art, the quantization method for the Vision Transformer neural network provided by the present application solves the problem that the existing ViT low-bit (four-bit quantization) quantization causes the classification accuracy to decrease in the image classification task.

[0008] In order to achieve the above-mentioned application purposes, the technical scheme adopted by the present application is:

[0009] The present application provides a quantization method for a Vision Transformer neural network, which comprises the following steps:

[0010] S1, training the Vision Transformer neural network by using an image classification data set to obtain a full-precision model;

[0011] S2, randomly selecting a preset number of pictures in the training set of the image classification data set to input the full-precision model, and statistically obtaining the maximum value and the minimum value of the weight and the activation value of each layer of the model as a quantization range;

[0012] S3, respectively quantizing the weight and the activation value of each layer according to the quantization range of the weight and the activation value of each layer;

[0013] S4, performing double-granularity quantization based on the Hessian matrix on the activation value after the Softmax, and performing scale reparameterization quantization based on channel optimization on the activation value of the layer normalization layer to obtain a quantized neural network model.

[0014] Further, the method for performing double-granularity quantization based on the Hessian matrix on the activation value after the Softmax comprises:

[0015] A1. Use the minimum mean square error to solve the maximum and minimum values ​​of the quantization range of the local optimal activation value after Softmax:

[0016]

[0017] in, and are the minimum and maximum values ​​of the local optimal quantization range respectively; A is the activation value after Softmax; For A The value after quantization; B is the activation value input to the subsequent matrix multiplication layer; for F norm;

[0018] A2. Expand the maximum and minimum values ​​of the local optimal quantization range by a preset ratio to obtain a new local optimal quantization range;

[0019] A3. Within the new local optimal quantization range, use the approximate Hessian matrix to solve the maximum and minimum values ​​of the global optimal quantization range of the activation value after Softmax:

[0020]

[0021] in, and are the minimum and maximum values ​​of the global optimal quantization range respectively; and They are A and B The output results of the product before and after quantization; is the Hessian matrix; diag () is a diagonal matrix; l For the l layer; and Respectively l layer O The first and second diagonal matrices of the matrix correspond to a elements; L Loss for the mission; T is transposed; min To take the minimum value;

[0022] A4. Perform asymmetric quantization on the activation value after Softmax according to the global optimal quantization range.

[0023] Further, A and B The output result after product quantization The expression is:

[0024]

[0025] wherein, is the quantized value of the l layer A ; is the activation value of the l layer input to the subsequent matrix multiplication layer.

[0026] Further, the method of performing channel-wise optimization based scale reparameterization quantization on the layer normalized layer activation value comprises:

[0027] B1, adopting minimum mean square error to optimize the layer normalized layer post-activation value and the reparameterized weight by channel:

[0028]

[0029] wherein, and are the minimum value and the maximum value of the quantization range of and respectively; is the F norm;

[0030] B2, according to the quantization range of and , respectively, performing asymmetric quantization on the layer normalized layer post-activation value and the reparameterized weight .

[0031] Further, the expression of the maximum value and the minimum value of the weight and the activation value of each layer is respectively:

[0032]

[0033] wherein, q 1 and q 2 are the maximum value and the minimum value of the weight or the activation value of each layer respectively; V is the tensor to be quantized; min and max are the minimum value and the maximum value respectively.

[0034] Further, the method of performing asymmetric quantization comprises:

[0035] According to the maximum value and the minimum value of the quantization range, calculating the scaling factor and the zero point:

[0036]

[0037] wherein, s and z are a scaling factor and a zero point; and are a minimum value and a maximum value of a quantization range, respectively; is a rounding function; b is a bit of a quantization target;

[0038] According to the scaling factor and the zero point, the floating-point value is asymmetrically quantized:

[0039]

[0040] wherein, x is a floating-point value, when the weights and the activation values in step S3 are asymmetrically quantized, the floating-point value is the weights and the activation values, respectively, when the post-Softmax activation value in step A4 is asymmetrically quantized, the floating-point value is the post-Softmax activation value, and when the and in step B2 are asymmetrically quantized, the floating-point value is and , respectively; is an integer value; is a value after the floating-point value x is quantized.

[0041] Further, the image classification dataset is an ImageNet dataset, and the preset number is 32.

[0042] The beneficial effects of the present application are: the present scheme performs double granularity quantization based on the Hessian matrix on the post-Softmax activation value, which can solve the long-tail distribution problem of the post-Softmax activation value and can also approximate the global optimal solution; in addition, the double granularity quantization based on the Hessian matrix can not only be used to quantize the post-Softmax activation value, but also can be used to quantize the activation values and weights of other layers, so that the ViT low-bit quantization can be efficiently deployed, and the classification accuracy of the quantized model can also be improved.

[0043] After the layer normalization layer activation value is quantized based on the channel optimization scale reparameterization, it can optimize the abnormal values in each channel of the post-activation value and the reparameterization weight of the layer normalization layer, so as to solve the problem of significant variance between channels, make the performance approach the per-channel quantization, meet the high performance demand under low-bit quantization, and further improve the classification accuracy of the quantized model. BRIEF DESCRIPTION OF DRAWINGS

[0044] Fig. 1 is a flowchart of a quantization method for a Vision Transformer neural network.

[0045] Fig. 2A comparison chart of inference time of the present scheme and existing multiple quantization methods. DETAILED DESCRIPTION

[0046] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0047] Reference Fig. 1 , Fig. 1 A flowchart of a quantization method for a Vision Transformer neural network is shown; as shown in Fig. 1 , the method S includes steps S1-S4.

[0048] In step S1, the Vision Transformer neural network is trained using an image classification dataset to obtain a full-precision model; wherein the image classification dataset is preferably an ImageNet dataset.

[0049] In step S2, a predetermined number of pictures are randomly selected from the training set of the image classification dataset and input into the full-precision model, and the predetermined number is preferably 32; the maximum and minimum values of the weights and activation values of each layer of the model are counted as the quantization range:

[0050]

[0051] wherein, q 1 and q 2 are the maximum and minimum values of the weights or activation values of each layer, respectively; V is the tensor to be quantized; min and max are the minimum and maximum values, respectively.

[0052] In step S3, according to the quantization range of the weights and activation values of each layer, the weights and activation values of the corresponding layer are respectively subjected to asymmetric quantization;

[0053] In step S4, the activation values after Softmax are subjected to double-granularity quantization based on the Hessian matrix, and the activation values of the layer normalization layer are subjected to scale reparameterization quantization based on channel optimization to obtain a quantized neural network model.

[0054] In an embodiment of the present application, the method of double-granularity quantization based on the Hessian matrix for the activation values after Softmax includes:

[0055] A1, the maximum and minimum of the locally optimal quantization range of the Softmax post-activation value are solved by using the minimum mean square error:

[0056]

[0057] wherein, and are the minimum and maximum of the locally optimal quantization range respectively; A is the Softmax post-activation value; is the value after quantization of A ; B is the activation value input to the subsequent matrix multiplication layer; is the F norm;

[0058] A2, the maximum and minimum of the locally optimal quantization range are expanded by a preset proportion to obtain a new locally optimal quantization range;

[0059] A3, the maximum and minimum of the globally optimal quantization range of the Softmax post-activation value are solved by using the approximate Hessian matrix in the new locally optimal quantization range:

[0060]

[0061] wherein, and are the minimum and maximum of the globally optimal quantization range respectively; and are the A and B output results before and after product quantization respectively; is the Hessian matrix; diag () is a diagonal matrix; l is the i-th layer; l and are the 1st and i-th elements of the i-th layer matrix corresponding to the diagonal matrix respectively; l is the task loss Loss; O is the transpose; a is the minimum value; L T A4, the Softmax post-activation value is asymmetrically quantized according to the globally optimal quantization range. min

[0062] A4, the Softmax post-activation value is asymmetrically quantized according to the globally optimal quantization range.

[0063] ​​In the double granularity quantization process based on the Hessian matrix, the scheme uses grid search for optimization, that is, A1 performs coarse-grained optimization, which locally minimizes the quantization error by finding a suitable quantization range. In order to further approach the global optimum, the scheme performs fine-grained optimization based on the local optimal quantization range found by coarse-grained optimization. The Hessian matrix is a second-order gradient, which contains more global information of the neural network, and the optimization of the quantization range through it can often approach the global optimum.

[0064] In implementation, the scheme preferably A and B the output result after product quantization The expression is:

[0065]

[0066] wherein, is the value after quantization of the first l layer A ; is the activation value of the first l layer input to the subsequent matrix multiplication layer.

[0067] In an embodiment of the present application, the method of scale reparameterization quantization based on channel optimization of layer normalization layer activation value comprises:

[0068] B1, using the least mean square error to optimize the layer normalization layer post-activation value channel by channel and the quantization range of the reparameterized weight :

[0069]

[0070] wherein, and are the minimum value and the maximum value of the quantization range of and ; is the F norm;

[0071] B2, according to the quantization range of and , respectively, asymmetrically quantize the layer normalization layer post-activation value and the reparameterized weight .

[0072] The scheme solves the problems of great deviation of the distribution of the activation value after the Softmax activation function from the Gaussian distribution and significant variance between channels of the activation value after the layer normalization layer by using the double granularity quantization based on the Hessian matrix for the activation value after the Softmax and using the scale reparameterization quantization based on channel optimization for the activation value after the layer normalization layer, thereby ensuring the classification accuracy of the quantized model.

[0073] In an embodiment of the present application, the method of performing asymmetric quantization comprises:

[0074] According to the maximum and minimum values of the quantization range, the scaling factor and the zero point are calculated:

[0075]

[0076] wherein, s and z are the scaling factor and the zero point; and are the minimum and maximum values of the quantization range, respectively; is the rounding function; b is the bit of the quantization target;

[0077] According to the scaling factor and the zero point, the floating point value is asymmetrically quantized:

[0078]

[0079] wherein, x is the floating point value, when the weights and the activation values in step S3 are asymmetrically quantized, the floating point value is the weights and the activation values, respectively, when the activation value after the Softmax in step A4 is asymmetrically quantized, the floating point value is the activation value after the Softmax, and when the and in step B2 are asymmetrically quantized, the floating point value is and , respectively; is the integer value; is the value after quantization of the floating point value x .

[0080] The effects of the quantization method of the present scheme on the classification task are described below in combination with multiple quantization methods in the prior art:

[0081] The multiple quantization methods introduced in this embodiment are:

[0082] PTQ4ViT introduces double uniform quantizers to handle the power-law distribution, and proposes a Hessian-guided search strategy for optimization. APQ-ViT enhances the Hessian-guided method through block-wise calibration, and adds a scale parameter in the quantization of attention maps to maintain the Matthew effect. TSPTQ-ViT proposes two-scale quantizers, V-2SF for the activation values after Softmax and GeLU layers, and O-2SF for the activation values after layer normalization layers. RepQ-ViT adopts scale reparameterization techniques to quantize the activation values after layer normalization layers to balance the larger activation quantization error and the smaller weight quantization error. It further introduces a log√2 quantizer to improve accuracy.

[0083] In addition to comparing the original version of the ViT model, this embodiment also compares two variants of the ViT model, DeiT (Data-efficient Image Transformer) is a variant of the ViT model designed to improve the performance of the ViT model with less data by introducing knowledge distillation. Swin Transformer (Shifted Window Transformer) is a variant of the ViT model designed to solve the efficiency problem of ViT and DeiT when processing large-scale images. Swin Transformer adopts a local window and hierarchical design to solve the high computational consumption problem of ViT while maintaining strong performance.

[0084] The quantization methods PTQ4ViT, APQ-ViT, TSPTQ-ViT, RepQ-ViT and the quantization method IFQ-ViT (ours) of the present scheme are applied to various ViT models (ViT-S, ViT-B, DeiT-T, DeiT-S, DeiT-B, Swin-S, Swin-B), and the quantization accuracy of the ViT model in the classification task on the ImageNet dataset is shown in Table 1.

[0085] Table 1 Quantization accuracy in classification task on ImageNet dataset

[0086]

[0087] In Table 1, Full-Precision represents the accuracy of the full-precision model, and IF represents whether the quantization scheme intervenes in inference.

[0088] From the comparison data in Table 1, it can be found that the quantization scheme proposed in the present scheme can achieve very high quantization accuracy, especially under four-bit quantization, the quantization accuracy is significantly improved.

[0089] When the quantization methods PTQ4ViT, RepQ-ViT and the quantization method IFQ-ViT of the present solution are applied to various ViT models (ViT-S, ViT-B, Swin-S, Swin-B), all methods run on an NVIDIA RTX 4090 graphics card, and the inference time of various ViT models can be referred to Fig. 2 . It can be known from Fig. 2 that the inference delay of the quantization method of the present solution is much lower than that of the PTQ4ViT and RepQ-ViT quantization methods of the display memory, which benefits from the fact that the quantization method of the present solution does not increase additional calculation and does not interfere with inference, so efficient inference can be performed.

[0090] In summary, the present solution quantizes the activation values after Softmax and the activation values of the layer normalization layer, so that the ViT low-bit quantization can be efficiently deployed, and at the same time, the ViT low-bit quantization can achieve higher classification accuracy.

Claims

1. A quantization method for Vision Transformer neural network, characterized in that Including steps: S1. Use the image classification dataset to train the Vision Transformer neural network to obtain a full-precision model; S2. Randomly select a preset number of images from the training set of the image classification dataset and input them into the full-precision model. Count the maximum and minimum values ​​of the weights and activation values ​​of each layer of the model as the quantization range. S3. According to the quantization range of the weights and activation values ​​of each layer, the weights and activation values ​​of the corresponding layers are asymmetrically quantized. Methods for performing asymmetric quantization include: Calculate the scaling factor and zero point based on the maximum and minimum values ​​of the quantization range: in, s and z is the scaling factor and zero point; and are the minimum and maximum values ​​of the quantization range respectively; is the rounding function; b is the bit number of the quantized target; Asymmetric quantization of floating-point values ​​according to a scaling factor and a zero point: in, x are floating-point values. When the weights and activation values ​​in step S3 are asymmetrically quantized, the floating-point values ​​are weights and activation values ​​respectively; S4. Performing dual-granularity quantization based on the Hessian matrix on the activation value after Softmax. The method of performing dual-granularity quantization based on the Hessian matrix on the activation value after Softmax includes: A1. Use the minimum mean square error to solve the maximum and minimum values ​​of the quantization range of the local optimal activation value after Softmax: in, and are the minimum and maximum values ​​of the local optimal quantization range respectively; A is the activation value after Softmax; For A The value after quantization; B is the activation value input to the subsequent matrix multiplication layer; for F norm; A2. Expand the maximum and minimum values ​​of the local optimal quantization range by a preset ratio to obtain a new local optimal quantization range; A3. Within the new local optimal quantization range, use the approximate Hessian matrix to solve the maximum and minimum values ​​of the global optimal quantization range of the activation value after Softmax: in, and are the minimum and maximum values ​​of the global optimal quantization range respectively; and They are A and B The output results of the product before and after quantization; is the Hessian matrix; diag () is a diagonal matrix; l For the l layer; and Respectively l layer O The first and second diagonal matrices of the matrix correspond to a elements; L Loss for the mission; T is transposed; min To take the minimum value; A4. Asymmetric quantization is performed on the activation value after Softmax according to the global optimal quantization range. When the activation value after Softmax in step A4 is asymmetric quantized, the floating-point value is the activation value after Softmax. The layer normalization layer activation value is subjected to scale reparameterization quantization based on channel optimization to obtain a quantized neural network model; the method for performing scale reparameterization quantization based on channel optimization on the layer normalization layer activation value includes: B1. Use the minimum mean square error to optimize the activation value after the normalization layer channel by channel and reparameterized weights Quantization range: in, and They are and The minimum and maximum values ​​of the quantization range; B2, according to and The quantization range of the layer is normalized and the activation value is and reparameterized weights Perform asymmetric quantization; when the and When asymmetric quantization is performed, the floating-point values ​​are and ; is an integer value; For floating point values x The quantized value.

2. The quantization method for a Vision Transformer neural network according to claim 1, characterized in that: A and B The output result after product quantization The expression is: in, For the l layer A The value after quantization; For the l The activation values ​​of the layer are input to the subsequent matrix multiplication layer.

3. The quantization method for a Vision Transformer neural network according to claim 1, wherein: The expressions for the maximum and minimum values ​​of the weights and activation values ​​of each layer are: in, q 1 and q 2 are the maximum and minimum values ​​of the weight or activation value of each layer respectively; V is the tensor to be quantized; min and max Take the minimum and maximum values ​​respectively.

4. The quantization method for a Vision Transformer neural network according to claim 1, wherein: The image classification dataset is the ImageNet dataset, and the preset number is 32.

Images