Efficient quantization method for Vision Transform neural network

Abstract

The invention discloses an efficient quantization method for a Vision Transform neural network, and the method comprises the steps: S1, carrying out the training of the Vision Transform neural network through employing an image classification data set, and obtaining a full-precision model; s2, randomly selecting a preset number of pictures in a training set of the image classification data set, inputting the pictures into a full-precision model, and counting the weight of each layer of the model and the maximum value and the minimum value of an activation value as a quantization range; s3, performing asymmetric quantization on the weight and the activation value of the corresponding layer according to the quantization range of the weight and the activation value of each layer; and S4, performing dual-granularity quantization based on a Hessian matrix on the activation value after Softmax, and performing scale re-parameterization quantization based on channel optimization on the activation value of the layer normalization layer to obtain a quantized neural network model.

Description

Technical Field

[0001] The present invention relates to a network quantization technology, and in particular to an efficient quantization method for a Vision Transformer neural network. Background Art

[0002] With the development of deep learning, many new neural network architectures have emerged, such as Transformer, Diffusion, etc. Inspired by the success of Transformer in natural language processing (NLP) tasks, researchers have extended its application to various computer vision (CV) tasks and developed a series of Vision Transformer (ViT) models. Now ViT has achieved advanced performance in image classification, object detection, semantic segmentation, etc. However, ViT relies on heavy and intensive calculations, which inevitably leads to a lot of memory usage, power consumption, and inference latency. These high overheads hinder the deployment of ViT on resource-constrained edge devices. Therefore, ViT's model compression is crucial in practical applications, especially in scenarios that require low-cost deployment and real-time inference.

[0003] Model quantization is an effective and popular model compression scheme. Model quantization converts high-precision weights and activation values ​​into low-precision values ​​through some kind of mapping, thereby reducing model complexity and computational complexity, making it more efficient to deploy in resource-constrained environments. In image classification tasks, there are now many studies on post-training quantization PTQ (PTQ) for ViT. They have identified two major challenges in quantizing ViT: the distribution of activation values ​​after the Softmax activation function deviates greatly from the Gaussian distribution, and the inter-channel variance of activation values ​​after the layer normalization layer is significant.

[0004] In order to solve the problem that the distribution of activation values ​​after the Softmax activation function deviates greatly from the Gaussian distribution, RepQ-ViT uses a Log√2 quantizer to quantize the activation values ​​after the Softmax activation function. The introduction of the Log√2 quantizer increases the computational load and latency during inference. This increase in computation makes such methods less suitable for hardware deployment, especially on resource-constrained devices. In addition, due to the large number of outliers in the activation value distribution, quantization of the activation values ​​after the Softmax activation function often results in large errors.

[0005] For the significant inter-channel variance of activation values ​​after the layer normalization layer, an ideal quantization solution is channel-wise quantization, but channel quantization hardware is incompatible and difficult to deploy. RepQ-ViT decouples the quantization and inference stages, and rescales and reparameterizes channel quantization to layer-wise quantization in the inference stage, so that the layer-wise quantization accuracy of activation values ​​after the layer normalization layer is close to channel-wise quantization, which is easier to deploy.

[0006] Although the method proposed by RepQ-ViT solves the problem of channel differences to a large extent by approximating channel-by-channel quantization through layer-by-layer quantization, there is still a certain gap compared with the performance of channel-by-channel quantization. Compared with channel-by-channel quantization, the scale reparameterization method will reduce the classification accuracy due to the presence of outliers in each channel of the activation value and reparameterization weight after the layer normalization layer. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the efficient quantization method for Vision Transformer neural network provided by the present invention solves the problem that the existing ViT low-bit (four-bit quantization) quantization will cause a decrease in classification accuracy in image classification tasks.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0009] An efficient quantization method for a Vision Transformer neural network is provided, comprising the steps of:

[0010] S1. Use the image classification dataset to train the Vision Transformer neural network to obtain a full-precision model;

[0011] S2. Randomly select a preset number of images from the training set of the image classification dataset and input them into the full-precision model, and count the maximum and minimum values ​​of the weights and activation values ​​of each layer of the model as the quantization range;

[0012] S3, according to the quantization range of the weight and activation value of each layer, asymmetric quantization is performed on the weight and activation value of the corresponding layer respectively;

[0013] S4. The activation values ​​after Softmax are quantized by double granularity based on the Hessian matrix, and the activation values ​​of the layer normalization layer are quantized by scale reparameterization based on channel optimization to obtain the quantized neural network model.

[0014] Furthermore, the method of performing double granularity quantization based on the Hessian matrix on the activation value after Softmax includes:

[0015] A1. Use the minimum mean square error to solve the maximum and minimum values ​​of the local optimal quantization range of the activation value after Softmax:

[0016]

[0017] Among them, q′ min and q′ max are the minimum and maximum values ​​of the local optimal quantization range respectively; A is the activation value after Softmax; is the quantized value of A; B is the activation value input to the subsequent matrix multiplication layer; ||·|| F is the F norm;

[0018] A2. Enlarging the maximum value and the minimum value of the local optimal quantization range by a preset ratio to obtain a new local optimal quantization range;

[0019] A3. Within the new local optimal quantization range, the approximate Hessian matrix is ​​used to solve the maximum and minimum values ​​of the global optimal quantization range of the activation value after Softmax:

[0020]

[0021] Among them, q″ min and q″ max are the minimum and maximum values ​​of the global optimal quantization range respectively; (l) and They are the output results of the product of A and B before and after quantization; is the Hessian matrix; diag() is the diagonal matrix; l is the lth layer; and They are the first and ath elements of the diagonal matrix corresponding to the O matrix of the lth layer respectively; L is the task loss Loss; T is the transpose; min is the minimum value;

[0022] A4. According to the global optimal quantization range, the activation value after Softmax is asymmetrically quantized.

[0023] Furthermore, the quantized output of the product of A and B is The expression is:

[0024]

[0025] in, is the quantized value of layer A; B (l) The activation value of the lth layer is input to the subsequent matrix multiplication layer.

[0026] Furthermore, the method of performing scale reparameterization quantization based on channel optimization on the layer normalization layer activation value includes:

[0027] B1, using the minimum mean square error to optimize the activation value X' and the weight W after the normalization layer channel by channel qkv Quantization range:

[0028]

[0029] Among them, q′ m ″ in and q′ m ″ ax X′ and W respectively qkv The minimum and maximum values ​​of the quantization range; is the F norm;

[0030] B2. According to X′ and W qkv The quantization range of the layer normalization layer activation value X′ and the re-parameterized weight W qkv Perform asymmetric quantization.

[0031] Furthermore, the expressions for the maximum and minimum values ​​of the weights and activation values ​​of each layer are:

[0032] q1=min(V),q2=max(V)

[0033] Among them, q1 and q2 are the maximum and minimum values ​​of the weight or activation value of each layer respectively; V is the tensor to be quantized; min and max are the minimum and maximum values ​​respectively.

[0034] Furthermore, the method for performing asymmetric quantization includes:

[0035] Calculate the scaling factor and zero point based on the maximum and minimum values ​​of the quantization range:

[0036]

[0037] Where s and z are the scaling factor and zero point; q min and q max are the minimum and maximum values ​​of the quantization range respectively; round(·) is the rounding function; b is the bit position of the quantization target;

[0038] Asymmetric quantization of floating-point values ​​according to a scaling factor and a zero point:

[0039]

[0040] Where x is a floating point value. When the weight and activation value in step S3 are asymmetrically quantized, the floating point values ​​are the weight and activation value respectively. When the activation value after Softmax in step A4 is asymmetrically quantized, the floating point value is the activation value after Softmax. When X′ and W in step B2 are asymmetrically quantized, the floating point value is the activation value after Softmax. qkv When asymmetric quantization is performed, the floating point values ​​are X′ and W respectively. qkv ;x int is an integer value; The floating point value x is quantized.

[0041] Furthermore, the image classification dataset is an ImageNet dataset, and the preset number is 32 images.

[0042] The beneficial effects of the present invention are as follows: the present scheme performs dual-granularity quantization on the activation value after Softmax based on the Hessian matrix, which can solve the problem of long-tail distribution of the activation value after Softmax and can approach the global optimal solution; in addition, the dual-granularity quantization based on the Hessian matrix can not only be used to quantize the activation value after Softmax, but also can quantize the activation values ​​and weights of other layers, so that ViT low-bit amount can be efficiently deployed, and at the same time can also improve the classification accuracy of the model after quantization.

[0043] After the activation values ​​of the layer normalization layer are quantized by scale reparameterization based on channel optimization, the outliers in each channel of the activation values ​​and reparameterization weights after the layer normalization layer can be optimized to solve the problem of significant variance between channels, so that the performance is close to channel-by-channel quantization, so as to meet the high performance requirements under low-bit quantization, so as to further improve the classification accuracy of the model after quantization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of an efficient quantization method for Vision Transformer neural networks.

[0045] Figure 2 This is a comparison chart of the inference time of this scheme and multiple existing quantization methods. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0047] refer to Figure 1 , Figure 1A flowchart of an efficient quantization method for Vision Transformer neural network is shown; Figure 1 As shown, the method S includes steps S1 to S4.

[0048] In step S1, the Vision Transformer neural network is trained using an image classification dataset to obtain a full-precision model; wherein the image classification dataset is preferably an ImageNet dataset.

[0049] In step S2, a preset number of pictures are randomly selected from the training set of the image classification data set and input into the full-precision model. The preset number is preferably 32 pictures. The maximum and minimum values ​​of the weights and activation values ​​of each layer of the statistical model are used as the quantization range:

[0050] q1=min(V),q2=max(V)

[0051] Among them, q1 and q2 are the maximum and minimum values ​​of the weight or activation value of each layer respectively; V is the tensor to be quantized; min and max are the minimum and maximum values ​​respectively.

[0052] In step S3, according to the quantization range of the weight and activation value of each layer, the weight and activation value of the corresponding layer are asymmetrically quantized;

[0053] In step S4, the activation values ​​after Softmax are quantized by double granularity based on the Hessian matrix, and the activation values ​​of the layer normalization layer are quantized by scale reparameterization based on channel optimization to obtain a quantized neural network model.

[0054] In one embodiment of the present invention, a method for performing dual granularity quantization based on a Hessian matrix on the activation value after Softmax includes:

[0055] A1. Use the minimum mean square error to solve the maximum and minimum values ​​of the local optimal quantization range of the activation value after Softmax:

[0056]

[0057] Among them, q′ min and q′ max are the minimum and maximum values ​​of the local optimal quantization range respectively; A is the activation value after Softmax; is the quantized value of A; B is the activation value input to the subsequent matrix multiplication layer; ||·|| F is the F norm;

[0058] A2. Enlarging the maximum value and the minimum value of the local optimal quantization range by a preset ratio to obtain a new local optimal quantization range;

[0059] A3. Within the new local optimal quantization range, the approximate Hessian matrix is ​​used to solve the maximum and minimum values ​​of the global optimal quantization range of the activation value after Softmax:

[0060]

[0061] Among them, q′ m ' in and q′ m ' ax are the minimum and maximum values ​​of the global optimal quantization range respectively; (l) and They are the output results of the product of A and B before and after quantization; is the Hessian matrix; diag() is the diagonal matrix; l is the lth layer; and They are the first and ath elements of the diagonal matrix corresponding to the O matrix of the lth layer respectively; L is the task loss Loss; T is the transpose; min is the minimum value;

[0062] A4. According to the global optimal quantization range, the activation value after Softmax is asymmetrically quantized.

[0063] In the dual-granularity quantization process based on the Hessian matrix, this scheme uses grid search for optimization, that is, A1 performs coarse-grained optimization. This step minimizes the quantization error locally by finding a suitable quantization range. In order to get closer to the global optimum, this scheme performs fine-grained optimization based on the local optimal quantization range found by coarse-grained optimization. The Hessian matrix is ​​a second-order gradient, which contains more global information of the neural network. Optimizing the quantization range through it can often approach the global optimum.

[0064] When implemented, this scheme prefers the output result of the quantized product of A and B The expression is:

[0065]

[0066] in, is the value after quantization of layer A; B (l) The activation value of the lth layer is input to the subsequent matrix multiplication layer.

[0067] In one embodiment of the present invention, a method for performing channel-optimized scale reparameterization quantization on layer normalization layer activation values ​​includes:

[0068] B1, using the minimum mean square error to optimize the activation value X' and the weight W after the normalization layer channel by channel qkv Quantization range:

[0069]

[0070] Among them, q′ m ″ in and q′ m ″ ax X′ and W respectively qkv The minimum and maximum values ​​of the quantization range; is the F norm;

[0071] B2. According to X′ and W qkv The quantization range of the layer normalization layer activation value X′ and the re-parameterized weight W qkv Perform asymmetric quantization.

[0072] This scheme uses dual-granularity quantization based on the Hessian matrix for the activation values ​​after Softmax, and uses scale reparameterization quantization based on channel optimization for the activation values ​​after the layer normalization layer. This solves the problem that the distribution of activation values ​​after the Softmax activation function deviates greatly from the Gaussian distribution, and the problem of significant inter-channel variance of activation values ​​after the layer normalization layer, thereby ensuring the classification accuracy of the quantized model.

[0073] In one embodiment of the present invention, the method for performing asymmetric quantization includes:

[0074] Calculate the scaling factor and zero point based on the maximum and minimum values ​​of the quantization range:

[0075]

[0076] Where s and z are the scaling factor and zero point; q min and q max are the minimum and maximum values ​​of the quantization range respectively; round(·) is the rounding function; b is the bit position of the quantization target;

[0077] Asymmetric quantization of floating-point values ​​according to a scaling factor and a zero point:

[0078]

[0079] Where x is a floating point value. When the weight and activation value in step S3 are asymmetrically quantized, the floating point values ​​are the weight and activation value respectively. When the activation value after Softmax in step A4 is asymmetrically quantized, the floating point value is the activation value after Softmax. When X′ and W in step B2 are asymmetrically quantized, the floating point value is the activation value after Softmax. qkv When asymmetric quantization is performed, the floating point values ​​are X′ and W respectively. qkv ;x int is an integer value; The floating point value x is quantized.

[0080] The following describes the effect of the efficient quantization method of this solution on the classification task in combination with multiple quantization methods in the prior art:

[0081] The multiple quantization methods introduced in this embodiment are:

[0082] PTQ4ViT introduces a dual uniform quantizer to handle power-law distribution and proposes a Hessian-guided search strategy for optimization. APQ-ViT enhances the Hessian-guided method by block-by-block calibration and adds a scale parameter to the quantization of the attention map to maintain the Matthew effect. TSPTQ-ViT proposes two scale quantizers, V-2SF for activation values ​​after the Softmax and GeLU layers, and O-2SF for activation values ​​after the layer normalization layer. RepQ-ViT uses scale reparameterization technology to quantize the activation values ​​after the layer normalization layer to balance the larger activation quantization error and the smaller weight quantization error. It also further introduces a log√2 quantizer to improve accuracy.

[0083] In addition to comparing the original version of the ViT model, this example also compares two variant models of ViT. DeiT (Data-efficient Image Transformer) is a ViT variant model that aims to improve the performance of ViT when the amount of data is small by introducing knowledge distillation. Swin Transformer (Shifted Window Transformer) is a ViT variant model that aims to solve the efficiency problems of ViT and DeiT when processing large-scale images. Swin Transformer adopts a local window and hierarchical design to solve the high computational consumption problem of ViT while maintaining strong performance.

[0084] The quantization methods PTQ4ViT, APQ-ViT, TSPTQ-ViT, RepQ-ViT and the efficient quantization method IFQ-ViT (ours) of this scheme are applied to various ViT models (ViT-S, ViT-B, DeiT-T, DeiT-S, DeiT-B, Swin-S, Swin-B). The quantization accuracy of the ViT model for classification tasks on the ImageNet dataset is shown in Table 1.

[0085] Table 1 Quantized accuracy of classification tasks on the ImageNet dataset

[0086]

[0087] In Table 1, Full-Precision indicates the full-precision model accuracy, and IF indicates whether the quantization scheme intervenes in reasoning.

[0088] From the comparative data in Table 1, it can be found that the quantization scheme proposed in this scheme can achieve very high quantization accuracy, especially under four-bit quantization, the quantization accuracy is significantly improved.

[0089] When applying the quantization methods PTQ4ViT, RepQ-ViT and the efficient quantization method IFQ-ViT of this solution to various ViT models (ViT-S, ViT-B, Swin-S, Swin-B), all methods are run on the NVIDIA RTX 4090 graphics card. The inference time of various ViT models can be referenced. Figure 2 .pass Figure 2 It can be seen that the inference delay of the efficient quantization method proposed in this scheme is much lower than the PTQ4ViT and RepQ-ViT quantization methods of the video memory. This is due to the fact that the efficient quantization method proposed in this scheme does not increase additional calculations and does not interfere with reasoning, so efficient reasoning can be performed.

[0090] In summary, this scheme enables efficient deployment of ViT low-bit quantization by quantizing the activation values ​​after Softmax and the activation values ​​of the layer normalization layer, and at the same time enables ViT low-bit quantization to achieve higher classification accuracy.

Claims

1. An efficient quantization method for Vision Transformer neural network, characterized in that: Includes steps: S1. Use the image classification dataset to train the Vision Transformer neural network to obtain a full-precision model. S2. Randomly select a preset number of images from the training set of the image classification dataset and input them into the full-precision model, and count the maximum and minimum values ​​of the weights and activation values ​​of each layer of the model as the quantization range; S3, according to the quantization range of the weight and activation value of each layer, asymmetric quantization is performed on the weight and activation value of the corresponding layer respectively; S4. The activation values ​​after Softmax are quantized by double granularity based on the Hessian matrix, and the activation values ​​of the layer normalization layer are quantized by scale reparameterization based on channel optimization to obtain the quantized neural network model.

2. The efficient quantization method according to claim 1, characterized in that: The method of double granularity quantization based on Hessian matrix for the activation value after Softmax includes: A1. Use the minimum mean square error to solve the maximum and minimum values ​​of the local optimal quantization range of the activation value after Softmax: Among them, q′ min and q′ max are the minimum and maximum values ​​of the local optimal quantization range respectively; A is the activation value after Softmax; is the quantized value of A; B is the activation value input to the subsequent matrix multiplication layer; ||·|| F is the F norm; A2. Enlarging the maximum value and the minimum value of the local optimal quantization range by a preset ratio to obtain a new local optimal quantization range; A3. Within the new local optimal quantization range, the approximate Hessian matrix is ​​used to solve the maximum and minimum values ​​of the global optimal quantization range of the activation value after Softmax: Among them, q′ m ' in and q′ m ' ax are the minimum and maximum values ​​of the global optimal quantization range respectively; (l) and They are the output results of the product of A and B before and after quantization; is the Hessian matrix; diag() is the diagonal matrix; l is the lth layer; and They are the first and ath elements of the diagonal matrix corresponding to the O matrix of the lth layer respectively; L is the task loss Loss; T is the transpose; min is the minimum value; A4. According to the global optimal quantization range, the activation value after Softmax is asymmetrically quantized.

3. The efficient quantization method according to claim 2, characterized in that: The output result after quantization of A and B The expression is: in, is the quantized value of layer A; B (l) The activation value of the lth layer is input to the subsequent matrix multiplication layer.

4. The efficient quantization method according to claim 1, characterized in that: The method of performing channel-optimized scale reparameterization quantization on the layer normalization layer activation value includes: B1, using the minimum mean square error to optimize the activation value X' and the weight W after the normalization layer channel by channel qkv Quantization range: Among them, q′ m ″ in and q′ m ″ ax X′ and W respectively qkv The minimum and maximum values ​​of the quantization range; is the F norm; B2. According to X′ and W qkv The quantization range of the layer normalization layer activation value X′ and the re-parameterized weight W qkv Perform asymmetric quantization.

5. The efficient quantization method according to claim 1, characterized in that: The expressions for the maximum and minimum values ​​of the weights and activation values ​​of each layer are: q1=min(V),q2=max(V) Among them, q1 and q2 are the maximum and minimum values ​​of the weight or activation value of each layer respectively; V is the tensor to be quantized; min and max are the minimum and maximum values ​​respectively.

6. The efficient quantization method according to any one of claims 1 to 5, characterized in that: Methods for performing asymmetric quantization include: Calculate the scaling factor and zero point based on the maximum and minimum values ​​of the quantization range: Where s and z are the scaling factor and zero point; q min and q max are the minimum and maximum values ​​of the quantization range respectively; round(·) is the rounding function; b is the bit position of the quantization target; Asymmetric quantization of floating-point values ​​according to a scaling factor and a zero point: Where x is a floating point value. When the weight and activation value in step S3 are asymmetrically quantized, the floating point values ​​are the weight and activation value respectively. When the activation value after Softmax in step A4 is asymmetrically quantized, the floating point value is the activation value after Softmax. When X″ and W in step B2 are asymmetrically quantized, the floating point value is the activation value after Softmax. qkv When asymmetric quantization is performed, the floating point values ​​are X′ and W respectively. qkv ;x int is an integer value; The floating point value x is quantized.

7. The efficient quantization method according to any one of claims 1 to 5, characterized in that: The image classification dataset is the ImageNet dataset, and the preset number is 32.

Images