A block-by-block post-training quantization method and system for ViTs

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically discloses a block-by-block post-training quantization method and system for ViTs, the method comprising: obtaining a full-precision Vision transformer pre-training model, calibration data and a quantizer; replacing all modules in the pre-training model with corresponding quantization modules to obtain a ViTs model to be quantized, wherein the quantization module includes a quantizer for activation values ​​and weights, and the activation value adopts a block-by-block uniform quantizer; using the calibration data, calculating the initial value of the scaling factor and the zero offset of each quantizer; using the calibration data and the pre-training model, iteratively optimizing the scaling factor to obtain a quantized ViTs model. The present invention allows each image block to have an independent quantization parameter, greatly improving the quantization resolution and making the quantized activation value have a stronger representation capability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image block-by-image block post-training quantization method and system for ViTs. Background Art

[0002] Vision Transformers (ViTs) have achieved excellent performance in many downstream computer vision tasks. However, the large number of parameters and storage requirements seriously hinder their deployment on resource-constrained hardware devices. In order to apply ViTs to practical applications, model quantization technology has received widespread attention from researchers. Model quantization is one of the most efficient and common model compression methods, which represents the full-precision model weights and / or activation values ​​in a lower-bit form.

[0003] Model quantization methods can be roughly divided into two major processes: Quantization Aware Training (QAT) and Post Training Quantization (PTQ). QAT uses the full dataset to restore the prediction accuracy of the quantized model by retraining or fine-tuning. Although the accuracy of the quantized model can be restored to the same level as the full-precision model, for large-scale models such as ViTs, the entire quantization process will be extremely time-consuming and energy-consuming. Recently, more and more researchers have turned to the study of PTQ methods. PTQ only requires a small unlabeled calibration dataset and no fine-tuning is required to quantize the pre-trained model, which greatly promotes the fast and efficient deployment of the model. The disadvantage is that the prediction accuracy of the ViTs model after post-training quantization will drop sharply, so most of the work focuses on how to restore the accuracy of the quantized model.

[0004] Most existing PTQ works of ViTs attribute the drop in accuracy to the inter-channel differences of post-LayerNorm activation values ​​and the non-uniform distribution of post-Softmax activation values, i.e., attention features. These special modules design a large number of sophisticated quantizers and quantization methods to eliminate the harmful effects of numerical differences on quantization and adapt to activation values ​​with special distributions. However, there are still two problems with existing methods: First, all post-LayerNorm activation values ​​with severe inter-channel differences share or partially share quantization parameters that are not important for quantization performance. This coarse-grained quantization reduces the quantization resolution and hinders the representation ability of activation values, resulting in increased quantization errors. Second, the optimization of quantization parameters related to attention features and the optimization of other quantization parameters are considered equally important, which may result in suboptimal attention feature quantization parameters and cause the quantized attention features to collapse. The collapsed attention features are difficult to accurately capture the global dependencies that constitute the inductive bias of ViTs, posing challenges to the performance of quantized ViTs.

[0005] The structure of the Vision transformer model is as follows Figure 1 As shown in the figure, the box plots and scatter plots of some channel values ​​of the input activation values ​​of some modules are as follows Figure 2 shown. Figure 2 (a)-(h) show that not only the post-LayerNorm activation value has serious inter-channel differences, but other activation values ​​also have varying degrees of numerical differences. There may also be a small number of outliers in each channel of the activation value. These varying degrees of numerical differences and outliers will have a negative impact on the quantization parameters, and coarse-grained quantization parameters will reduce the quantization resolution. On the one hand, it is difficult to find a common quantization parameter to adapt to all channels. Coarse-grained quantization parameters mean that an integer value in the quantization domain corresponds to a larger numerical interval in the floating-point domain. On the other hand, outliers are likely to dominate the calculation of the quantization parameters. These outliers seriously deviate from other values ​​and cannot fully utilize the quantization level, i.e., the integer value, in the quantization domain. Therefore, it is difficult for existing PTQ methods to achieve satisfactory accuracy, especially in low-bit situations. Summary of the invention

[0006] The present invention aims to solve the problem that the quantization parameters of ViTs activation values ​​are seriously affected by different degrees of inter-channel numerical differences and outliers, and the coarse-grained quantization parameters lead to unsatisfactory quantization resolution. To this end, the present invention provides a block-by-block post-training quantization method and system for ViTs, which allows each image block to have independent quantization parameters, greatly improves the quantization resolution and makes the quantized activation values ​​have stronger representation capabilities.

[0007] The present invention provides a block-by-block post-training quantization method for ViTs, and the technical solution adopted is as follows: comprising the following steps:

[0008] Step 1: Get the full-precision Vision transformer pre-trained model, calibration data, and quantizer;

[0009] Step 2: Replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain the ViTs model to be quantized, wherein the quantization module includes a quantizer of an activation value and a weight, and the activation value adopts a uniform quantizer per image block;

[0010] Step 3: Using the calibration data, calculate the initial value of the scaling factor and zero offset of each quantizer in the ViTs model to be quantized;

[0011] Step 4: Using the calibration data and the full-precision Vision transformer pre-trained model, iteratively optimize the scaling factor of the ViTs model to be quantized to obtain a quantized ViTs model.

[0012] Furthermore, the activation value of the FC(QKV) module, the activation value Q of the MatMul1 module, the activation value K of the MatMul1 module, the activation value of the FC(Out) module, the activation value of the FC1 module and the activation value of the FC2 module adopt a per-image block uniform quantizer.

[0013] Furthermore, the calculation formula of the uniform quantizer for each image block is:

[0014]

[0015] Among them, x q Represents the quantized activation value matrix, Clamp() represents the truncation function, Round() represents the rounding function, diag() represents the diagonalization function, s represents the scaling factor of each image block, x represents the activation value matrix, zp represents the zero point offset of each image block, 1 represents the full 1 row vector whose number of elements is equal to the number of columns of the activation value matrix x, and b represents the quantization bit width.

[0016] Furthermore, the weights of all the quantization modules in the ViTs model to be quantized and the activation value V of the MatMul2 module adopt a channel-by-channel uniform quantizer; the activation value Att of the MatMul2 module adopts a logarithmic quantizer.

[0017] Furthermore, in step 3, the scaling factor and zero offset of each image block uniform quantizer are calculated using the percentile method, and the calculation formula is:

[0018]

[0019]

[0020] in, express The corresponding scaling factor is, express The corresponding zero point offset, percentile() represents the percentile calculation function, Represents the activation value matrix The image blocks, represents percentile, Round() represents rounding function, and b represents quantization bit width.

[0021] Furthermore, the percentiles include 0.999, 0.9999 and 0.99999,

[0022] The percentile selection steps include:

[0023] For each image block, the scaling factor and the zero point offset are calculated according to the percentiles, and then the losses are calculated respectively, and the percentile corresponding to the minimum loss is selected as the percentile corresponding to the image block.

[0024] Furthermore, in step 4, the scaling factors in the convolutional layer of the input part and the scaling factors in the linear layer of the output part of the ViTs model to be quantized are optimized layer by layer, and the remaining scaling factors are optimized by stacked encoding modules.

[0025] Furthermore, the specific process of iterative optimization is:

[0026] Traversing each quantization module in the ViTs model to be quantized in turn;

[0027] If the quantization module is a convolutional layer or an output linear layer, the calibration data is input into the corresponding convolutional layer or output linear layer in the full-precision Visiontransformer pre-trained model to obtain the input activation value and the output activation value; then the iterative optimization is started, and each iteration inputs part of the samples in the input activation value into the quantization module to obtain the dequantization result of the output activation value, and the loss between the dequantization result and the output activation value is calculated. The scaling factor is back-propagated and optimized using the loss to finally obtain the optimized scaling factor;

[0028] If the quantization module is an encoding module, the calibration data is input into the corresponding encoding module in the full-precision Vision transformer pre-trained model to obtain the input activation value, the attention feature and the output activation value; then the iterative optimization is started, and in each iteration, part of the samples in the input activation value are input into the quantization module to obtain the inverse quantization result of the attention feature and the inverse quantization result of the output activation value, and the loss between them and the attention feature and the output activation value is calculated respectively, and the scaling factor is back-propagated and optimized using the loss to finally obtain the optimized scaling factor.

[0029] Furthermore, the loss of layer-wise optimization is:

[0030]

[0031] in, represents the mean square error loss, represents the mean operation, represents the square of the L2 norm, Indicates The output activation value of each module is Indicates The dequantization result of the output activation value of each module;

[0032] The loss optimized by the stacked encoding module is:

[0033]

[0034] in, Indicates The total loss of the module is represents the balance factor, represents the loss of the attention-weighted metric,

[0035]

[0036] Indicates The attention features of each module, Indicates The dequantization result of the attention feature of the module, |·| represents the absolute value operation, Represents the Hadamard product.

[0037] The present invention also provides a block-by-block post-training quantization system for ViTs, which adopts the following technical solution: comprising: an acquisition module, a calibration module and an optimization module connected in sequence,

[0038] The acquisition module is used to acquire a full-precision Vision transformer pre-trained model, calibration data, and a quantizer;

[0039] The calibration module is used to replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain a ViTs model to be quantized, wherein the quantization module includes a quantizer of an activation value and a weight, and the activation value adopts a uniform quantizer per image block; using the calibration data, the scaling factor and the initial value of the zero offset of each quantizer in the ViTs model to be quantized are calculated;

[0040] The optimization module is used to iteratively optimize the scaling factor of the ViTs model to be quantized by using the calibration data and the full-precision Vision transformer pre-training model to obtain a quantized ViTs model.

[0041] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0042] The present invention designs a block-by-block quantization, and independently calculates quantization parameters for each block of activation values ​​after filtering out outliers to support pure integer matrix multiplication on hardware during inference. In order to obtain better scaling factors related to attention features in the optimization stage, the present invention also designs an attention-weighted metric aweMSE to ensure that the inverse quantized attention features are similar to the full-precision attention features and further reduce the quantization error. The present invention alleviates the negative impact of severe numerical differences and outliers from the image block dimension of ViTs activation values, thereby improving the quantization resolution and generating more representative quantized activation values.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a structural diagram of the Vision transformer model in the prior art.

[0046] Figure 2 It is a box plot and a scatter plot of some channel values ​​of the input activation values ​​of some modules of the Vision transformer model provided by the present invention.

[0047] Figure 3 It is a flow chart of the method provided by the present invention.

[0048] Figure 4 This is the quantization resolution comparison result of W4A4 quantization provided by the present invention Figure 1 .

[0049] Figure 5 This is the quantization resolution comparison result of W4A4 quantization provided by the present invention Figure 2 . DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0051] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0052] Most existing PTQ works of ViTs share or partially share quantization parameters that are not important for quantization performance for all post-LayerNorm activation values ​​with severe inter-channel differences. The optimization of quantization parameters related to attention features and the optimization of other quantization parameters are considered equally important. The existing PTQ of ViTs is a coarse-grained quantization. To this end, the present invention first re-examines the characteristics of the activation values ​​of ViTs and observes that the activation values ​​of ViTs have different degrees of inter-channel differences, not just limited to the post-LayerNorm activation values; and there may be a small number of outliers within each channel. The calculation of quantization parameters is easily affected by these characteristics. An instinctive idea to solve the above coarse-grained quantization problem is to calculate the quantization parameters for each channel separately. Unfortunately, this approach cannot support pure integer matrix multiplication on hardware. Therefore, the present invention attempts to design a fine-grained PTQ method for ViTs from the image block dimension of the activation value rather than the channel dimension. Specifically, the present invention calculates the quantization parameters for each image block in the activation value after filtering out the outliers to alleviate the impact of severe numerical differences and outliers. The present invention proposes quantization per image block, which not only makes full use of the quantization level in the quantization domain, but also can efficiently reduce the overhead of floating-point and integer conversion on hardware through pure integer matrix multiplication during inference. In addition, the present invention also designs an attention-weighted metric aweMSE to optimize the quantization parameters related to the attention feature, which can make the quantized attention feature as close as possible to the original attention feature during optimization. In this way, the ability of quantized ViTs to capture global dependencies can be further maintained.

[0053] Combine the following Figures 3 to 5 The present invention is further described in detail, and a block-by-block post-training quantization method and system for ViTs of the present invention are described:

[0054] In this embodiment, Figure 3 As shown, a block-by-block post-training quantization method for ViTs is provided, comprising the following steps:

[0055] Step 1: Get the full-precision Vision transformer pre-trained model, calibration data, and quantizer.

[0056] Step 2: Replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain the ViTs model to be quantized. The quantization module includes quantizers for activation values ​​and weights, and the activation value adopts a uniform quantizer for each image block.

[0057] This embodiment uses different quantizers for activation values ​​and weights of different modules.

[0058] The present invention designs a block-by-block quantization method. Quantization is performed from the image block dimension in order to improve the quantization resolution in a fine-grained manner while enabling it to support pure integer matrix multiplication during hardware inference.

[0059] The calculation formula of the uniform quantizer for each image block is:

[0060]

[0061] Among them, x q represents the quantized activation matrix, Clamp() represents the truncation function, Round() represents the rounding function, diag() represents the diagonalization function, s represents the scaling factor of each image block, x represents the activation matrix, zp represents the zero offset of each image block, 1 represents the full 1 row vector, the number of elements of which is equal to the number of columns of the activation matrix x, and b represents the quantization bit width. Both s and zp are column vectors and the number of elements is equal to the number of rows (number of image blocks) of the activation matrix x.

[0062] The expression of supporting pure integer matrix multiplication characteristics brought by image block quantization is:

[0063]

[0064]

[0065] The activation value matrix of the result is The scaling factor corresponding to the row element, The first image blocks, The activation value matrix of the result is The zero offset corresponding to the row element, The activation value matrix of the result is Line The quantized values ​​of the column elements, Represents the first The scaling factor corresponding to the row elements, Represents the first The zero offset corresponding to the row element, Represents the first Line The quantized values ​​of the column elements, The weight matrix The scaling factor corresponding to the column elements, The weight matrix The zero offset corresponding to the column elements, The weight matrix Line The quantized values ​​of the column elements, Represents the common factor extracted using the scaling factor. The resulting activation value matrix is ​​the product of the floating-point activation value matrix and the weight matrix. , and All are floating point values.

[0066] Fixed-point multiplication and shift operations can be used for offline calculations. The calculation formula only contains integer operations. Therefore, the overhead of floating-point and integer conversion on the hardware can be greatly reduced. However, the fine-grained quantization on the channel dimension used in the existing technology cannot extract common factors, and cannot support integer matrix operations on the hardware.

[0067] In this embodiment, the activation value of the FC(QKV) module, the activation value Q of the MatMul1 module, the activation value K of the MatMul1 module, the activation value of the FC(Out) module, the activation value of the FC1 module, and the activation value of the FC2 module adopt a block-by-block uniform quantizer. The above activation values ​​are quantized block by block using a block-by-block uniform quantizer.

[0068] In this embodiment, the weights of all the quantization modules in the ViTs model to be quantized and the activation value V of the MatMul2 module adopt a channel-by-channel uniform quantizer; the activation value Att of the MatMul2 module adopts a logarithmic quantizer. The channel-by-channel uniform quantizer and the logarithmic quantizer are commonly used quantizers and belong to the prior art, which will not be described here.

[0069] Step 3: Using the calibration data, calculate the initial value of the scaling factor and zero offset of each quantizer in the ViTs model to be quantized.

[0070] The specific process is: the previous batch size in the calibration data Samples are input into the ViTs model to be quantized, and the quantization parameters, quantization parameter scaling factors and zero point offsets in the quantizers corresponding to each module in the ViTs model to be quantized are calculated to complete the initialization of the quantization parameters of the activation values ​​and weights.

[0071] In this process, the process of calculating the quantization parameters for each image block is as follows:

[0072] The scaling factor and zero offset of each per-block uniform quantizer are calculated using the percentile method as follows:

[0073]

[0074]

[0075] in, express The corresponding scaling factor is, express The corresponding zero point offset, percentile() represents the percentile calculation function, Represents the activation value matrix The image blocks, represents percentile, Round() represents rounding function, and b represents quantization bit width.

[0076] This embodiment uses the percentile method instead of the maximum and minimum values ​​to filter out abnormal values ​​and determine the quantization threshold of each image block. From multiple percentiles, one is selected according to the quantization errors caused by them to determine the upper and lower limits of the floating point value.

[0077] The percentile selection steps include:

[0078] For each image block, the scaling factor and the zero offset are calculated according to the percentiles, and then the loss is calculated, and the percentile corresponding to the minimum loss is selected as the percentile corresponding to the image block. The loss in this implementation adopts the mean square error loss.

[0079] Illustratively, the percentiles include 0.999, 0.9999, and 0.99999.

[0080] In this embodiment, for each image block, the one with the smallest corresponding loss is selected from 0.999, 0.9999 and 0.99999 as the percentile of the image block, and then the scaling factor and the zero offset are calculated.

[0081] Step 4: Using the calibration data and the full-precision Vision transformer pre-trained model, iteratively optimize the scaling factor of the ViTs model to be quantized to obtain a quantized ViTs model.

[0082] In step 4, the scaling factors are iteratively optimized in units of stacked coding modules (blocks) or layers. The scaling factors in the convolutional layer of the input part of the ViTs model to be quantized and the scaling factors in the linear layer of the output part are optimized by layer, and the remaining scaling factors are optimized by stacked coding modules, that is, the scaling factors of all modules within a coding module are optimized simultaneously.

[0083] The specific process of iterative optimization is:

[0084] Each quantization module in the ViTs model to be quantized is traversed in sequence.

[0085] If the quantization module is a convolutional layer or an output linear layer, the scaling factor is optimized layer by layer. Specifically, the calibration data is input into the corresponding convolutional layer or output linear layer in the full-precision Vision transformer pre-trained model to obtain the input activation value and the output activation value; then iterative optimization is started, and each iteration converts the sampling batch size of the input activation value into The samples are input into the quantization module to obtain the inverse quantization result of the output activation value, the loss between the inverse quantization result and the output activation value is calculated, and the scaling factor is back-propagated and optimized using the loss to finally obtain the optimized scaling factor.

[0086] The loss for layer-wise optimization is:

[0087]

[0088] in, represents the mean square error loss, represents the mean operation, represents the square of the L2 norm, Indicates The output activation value of each module is Indicates The dequantization result of the output activation value of each module.

[0089] If the quantization module is an encoding module, the scaling factor is optimized according to the stacked encoding modules. Specifically, the calibration data is input into the corresponding encoding module in the full-precision Vision transformer pre-trained model to obtain the input activation value, attention features, and output activation value; then iterative optimization is started, and each iteration samples the batch size of the input activation value. 1 sample is input into the quantization module to obtain the inverse quantization results of the attention feature and the output activation value, and the losses between them and the attention feature and the output activation value are calculated respectively. The scaling factor is back-propagated and optimized using the losses to finally obtain the optimized scaling factor.

[0090] The loss optimized by the stacked encoding module is:

[0091]

[0092] in, Indicates The total loss of the module is represents the loss of the attention-weighted metric, Represents the balance factor, which is used to balance the ratio of the loss of the mean squared error loss and the attention weighted metric. Its value is empirically set to 1e-5.

[0093]

[0094] Indicates The attention features of each module, Indicates The dequantization result of the attention feature of the module, |·| represents the absolute value operation, Represents the Hadamard product.

[0095] The loss of the attention-weighted metric is a loss specially designed by the present invention, which is used to emphasize the optimization of the scaling factor associated with the attention feature during block-by-block reconstruction. The loss makes the dequantized attention feature as close as possible to the original attention feature by adjusting the scaling factor associated with the attention feature. Since the attention feature essentially reflects the correlation between image blocks, the loss of the attention-weighted metric can represent the relative importance of each item in the L2 norm result.

[0096] After the traversal is completed, the optimization of the scaling factors of the activation values ​​and weights is completed, and the final quantized ViTs model is obtained.

[0097] Experiments: This embodiment (PTW-ViT) is compared with the existing PTQ method based on multiple ViT variant models on image classification, object detection, and instance segmentation tasks. Finally, ablation experiments of per-image block quantization and attention weighted metrics are performed to verify their effectiveness.

[0098] Experimental setup: PTW-ViT is implemented in PyTorch framework with pre-trained vision transformer variants such as ViT-S, ViT-B, DeiT-T, DeiT-S, DeiT-B, Swin-S and Swin-B from timm library. All experiments are performed on an NVIDIA A800 GPU. We evaluate PTW-ViT on image classification, object detection and instance segmentation tasks based on ImageNet1K and COCO datasets. 1024 and 1 samples are randomly selected from ImageNet1K training set and COCO training set respectively for calibrating quantization parameters. The full test set is used to evaluate performance. Adam optimizer is used to optimize the scaling factor. The initial learning rate is set to 4e-5 and the cosine annealing scheduler is used. Straight Through Estimator (STE) is used for non-differentiable rounding functions. In the image classification task, the batch size is set to 64, the number of optimization iterations for 4 bits is 300, and the number of iterations for 6 bits is 60. In the object detection and instance segmentation tasks, the Swin backbone network of the Mask R-CNN and Cascade Mask R-CNN frameworks is optimized in the same way as in the method, with the batch size set to 1, the number of optimization iterations for 4 bits is 50, and the number of optimization iterations for 6 bits is 5.

[0099] Table 1 Experimental results based on ImageNet dataset

[0100]

[0101] Table 1 compares the Top-1 accuracy (%) of the PTQ method on the Vision transformer variant and the full-precision model on the image classification task based on the ImageNet1K dataset. FQ-ViT, PTQ4ViT, APQ-ViT, RepQ-ViT, Ranking-ViT and NoisyQuant are existing PTQ methods. Bit width (W / A) represents the quantization bit width of activation values ​​and weights, in bits.

[0102] As shown in Table 1, under the W6A6 quantization bit width, the top-1 accuracy of this method exceeds the existing advanced PTQ methods on multiple visiontransformer variant models. Specifically, the performance improvements on ViT-B, DeiT-T, DeiT-S, DeiT-B and Swin-B are 0.22%, 0.58%, 0.13%, 0.08% and 0.34%, respectively. The average performance improvement on these variant models is nearly 0.27%. In addition, in the case of W4A4, this method exceeds the advanced methods by 3.19%, 4.48%, 2.46%, 1.06%, 0.87%, 0.60% and 3.28%, respectively, and the average performance improvement reaches a significant 2.28%, which fully demonstrates the superiority of this method in the case of lower precision. In order to more intuitively show that PTW-ViT can produce more expressive attention features, Figure 4 and Figure 5 Comparison results of quantization resolution in the W4A4 case are shown.

[0103] like Figure 4 As shown, Figure 4 (d) Visualizes a full-precision input activation value tensor of the FC(QKV) module of the ViT-B model. The shades of color represent different values. Figure 4 (a) shows an activation value slice and its dequantization results after using different quantization methods in W4A4. Figure 4 (b) is the result of dequantization of activation slices using tensor quantization, which calculates a common quantization parameter for the entire activation value with numerical differences and outliers. Figure 4 As can be seen in (b), there are many blocks with the same color, indicating that they are the same inverse quantized values. The same color or value means that these inverse quantized values ​​come from the same integer value in the quantization domain. In other words, Figure 4 The result in (b) shows that tensor quantization reduces the quantization resolution of the activation value slice. Figure 4 (c) is the dequantization result of the activation value slice using the block-by-block quantization proposed by the present invention. Obviously, the dequantization result of this method has richer colors and retains more representation information in the activation value slice. Figure 4 (b) Figure 4 The results in (c) are also consistent with Figure 4 The original activation value slices in (a) are more similar, indicating that the fine-grained quantization method of the present invention can effectively improve the quantization resolution and generate more representative quantized activation values.

[0104] Figure 5 The comparison results of a larger range of activation value areas at W4A4 are shown in Figure 2, and the same conclusion can still be drawn. Figure 5The contents of (a)-(c) are the same as Figure 4 (a)-(c) correspond to each other. Figure 4 and Figure 5 The difference is: Figure 4 The 5×5 matrix is ​​intercepted and the specific values ​​are displayed. Figure 5 The matrix with larger range (30×50 size) and undisplayed values ​​is captured.

[0105] Table 2 Experimental results based on COCO dataset

[0106]

[0107] Table 2 shows the AP of the PTQ method and the full-precision model on the object detection and instance segmentation tasks based on the COCO dataset box and AP mask (%) Comparison results. AP box Represents the average precision of the detection box in the target detection task, AP mask Represents the average precision of masks in the instance segmentation task.

[0108] As shown in Table 2, the two Swin transformer models w.Swin-T and w.Swin-S are used as the backbone network in the Mask R-CNN and Cascade Mask R-CNN frameworks. Note that since the number of image blocks in these two tasks changes with the change of the input image resolution, this embodiment only applies the per-image block quantization method to the MatMul1 module. In the case of W4A4, the average accuracy of the PTW-ViT method on Mask R-CNN and Cascade MaskR-CNN with w.Swin-T as the backbone network is 38.3% AP respectively. box and 37.0% AP mask , significantly improving by 2.2% and 1.0%. In addition, PTW-ViT achieves competitive results in the W6A6 case, with a small drop in accuracy compared to the corresponding full-precision model.

[0109] Table 3 Ablation experiment results of per-image block quantization method and attention weighted metric

[0110]

[0111] Ablation experiment: In order to verify the effectiveness of the block-by-block quantization and attention-weighted metric proposed in the present invention, ablation experiments were performed on image classification tasks based on the ViT-B and DeiT-T models. The experimental results are shown in Table 3. The baseline model performs coarse-grained tensor quantization for activation values, performs channel-by-channel quantization for weights, and only uses reconstruction loss to optimize the scaling factor. Patch Token Wise Quant. indicates that the block-by-block quantization method is used for the baseline model; AttentionWeighted Metric indicates that the attention-weighted metric is used for the baseline model; PTW-ViT (Ours) indicates this method, that is, the block-by-block quantization method and attention-weighted metric are used for the baseline model at the same time. For example, it can be seen from the experimental results of DeiT-T that the accuracy of the baseline model is severely reduced when W4A4 is quantized. After applying block-by-block quantization, the top-1 accuracy is significantly improved by 5.13%, showing the effectiveness and necessity of using the fine-grained quantization method of the present invention for the activation values ​​of ViTs. This also shows that improving the quantization resolution and enhancing the representation ability of the activation values ​​after quantization bring benefits to quantization. Introducing the attention-weighted metric aweMSE into the loss can further improve the performance in block-by-block reconstruction, showing that the joint loss can indeed contribute to the function of maintaining the quantized attention features.

[0112] Table 4 Ablation experiment results of different activation values ​​using per-image block quantization method

[0113]

[0114] Table 4 shows the ablation experiment of different activation values ​​using the per-image block quantization method in the case of W4A4. “√” indicates that the per-image block quantization method is applied to the corresponding module. The “√” below the module marked by “” indicates that the channel-by-channel quantization method is adopted.

[0115] Table 4 shows the results of using patch-wise quantization for each module in DeiT-T. Note that when each module does not use patch-wise quantization at all, the model is the baseline model. Obviously, as patch-wise quantization is applied to the corresponding modules, the accuracy is improved. Note that when patch-wise quantization is applied to the FC2 module, the accuracy is consistent with the baseline model accuracy (the last row in Table 4). In order to prove that patch-wise quantization does play a role in this module, patch-wise quantization is applied to other modules except the FC2 module (the second to last row in Table 4) for verification. The results in Table 4 also show that the activation values ​​in ViTs should use patch-wise quantization to alleviate the impact of severe numerical differences and outliers to reduce the accumulation of quantization errors, rather than just post-LayerNorm activation values.

[0116] This method is a fine-grained and hardware-friendly post-training quantization method for Vision transformers. It improves the quantization resolution and generates more representative quantized activation values ​​by alleviating the negative impact of severe numerical differences and outliers from the image block dimension of ViTs activation values. Specifically, the per-image block quantization independently calculates the quantization parameters for each image block of activation values ​​after filtering out outliers to support pure integer matrix multiplication on hardware during inference. In addition, in order to obtain better scaling factors related to attention features in the optimization stage, this method also designs an attention-weighted metric aweMSE to ensure that the dequantized attention features are close to the full-precision attention features and further reduce the quantization error. Extensive experiments verify that the performance of this method exceeds existing methods on image classification tasks, even in low bit width cases. PTW-ViT also achieves competitive performance on object detection and instance segmentation tasks.

[0117] This embodiment also provides a block-by-block post-training quantization system for ViTs, which adopts the following technical solution: comprising: an acquisition module, a calibration module and an optimization module connected in sequence.

[0118] The acquisition module is used to acquire a full-precision Vision transformer pre-trained model, calibration data, and a quantizer;

[0119] The calibration module is used to replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain a ViTs model to be quantized, wherein the quantization module includes a quantizer of an activation value and a weight, and the activation value adopts a uniform quantizer per image block; using the calibration data, the scaling factor and the initial value of the zero offset of each quantizer in the ViTs model to be quantized are calculated;

[0120] The optimization module is used to iteratively optimize the scaling factor of the ViTs model to be quantized by using the calibration data and the full-precision Vision transformer pre-training model to obtain a quantized ViTs model.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A block-by-block post-training quantization method for ViTs, characterized in that: The following steps are involved: Step 1: Get the full-precision Vision transformer pre-trained model, calibration data, and quantizer; Step 2: Replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain the ViTs model to be quantized, wherein the quantization module includes a quantizer of an activation value and a weight, and the activation value adopts a uniform quantizer per image block; The calculation formula of the uniform quantizer for each image block is: Among them, x q represents the quantized activation value matrix, Clamp() represents the truncation function, Round() represents the rounding function, diag() represents the diagonalization function, s represents the scaling factor of each image block, x represents the activation value matrix, zp represents the zero point offset of each image block, 1 represents the full 1 row vector, the number of its elements is equal to the number of columns of the activation value matrix x, and b represents the quantization bit width; Step 3: Using the calibration data, calculate the initial value of the scaling factor and zero offset of each quantizer in the ViTs model to be quantized; Step 4: Using the calibration data and the full-precision Vision transformer pre-trained model, iteratively optimize the scaling factor of the ViTs model to be quantized to obtain a quantized ViTs model.

2. A block-by-block post-training quantization method for ViTs according to claim 1, characterized in that: The activation value of the FC(QKV) module, the activation value Q of the MatMul1 module, the activation value K of the MatMul1 module, the activation value of the FC(Out) module, the activation value of the FC1 module and the activation value of the FC2 module adopt a per-image block uniform quantizer.

3. A block-by-block post-training quantization method for ViTs as claimed in claim 2, characterized in that: The weights of all the quantization modules in the ViTs model to be quantized and the activation value V of the MatMul2 module adopt a channel-by-channel uniform quantizer; the activation value Att of the MatMul2 module adopts a logarithmic quantizer.

4. A block-by-block post-training quantization method for ViTs as claimed in claim 1 or 2, characterized in that: In step 3, the scaling factor and zero offset of each per-block uniform quantizer are calculated using the percentile method, as follows: in, express The corresponding scaling factor is, express The corresponding zero point offset, percentile() represents the percentile calculation function, Represents the activation value matrix The image blocks, represents percentile, Round() represents rounding function, and b represents quantization bit width.

5. The image block-by-image block post-training quantization method for ViTs according to claim 4, characterized in that: The percentiles include 0.999, 0.9999, and 0.99999, The percentile selection steps include: For each image block, the scaling factor and the zero point offset are calculated according to the percentiles, and then the losses are calculated respectively, and the percentile corresponding to the minimum loss is selected as the percentile corresponding to the image block.

6. The image block-by-image block post-training quantization method for ViTs according to claim 1, characterized in that: In step 4, the scaling factors in the convolutional layer of the input part and the scaling factors in the linear layer of the output part of the ViTs model to be quantized are optimized layer by layer, and the remaining scaling factors are optimized by stacked encoding modules.

7. A block-by-block post-training quantization method for ViTs as claimed in claim 6, characterized in that: The specific process of iterative optimization is: Traversing each quantization module in the ViTs model to be quantized in turn; If the quantization module is a convolutional layer or an output linear layer, the calibration data is input into the corresponding convolutional layer or output linear layer in the full-precision Vision transformer pre-trained model to obtain the input activation value and the output activation value; Then iterative optimization begins. In each iteration, part of the samples in the input activation value are input into the quantization module to obtain the dequantization result of the output activation value, and the loss between the dequantization result and the output activation value is calculated. The scaling factor is back-propagated and optimized using the loss to finally obtain the optimized scaling factor. If the quantization module is an encoding module, the calibration data is input into the corresponding encoding module in the full-precision Vision transformer pre-trained model to obtain input activation values, attention features, and output activation values; Then the iterative optimization starts. In each iteration, part of the samples in the input activation value are input into the quantization module to obtain the inverse quantization results of the attention feature and the inverse quantization results of the output activation value. The losses between them and the attention feature and the output activation value are calculated respectively. The scaling factor is back-propagated and optimized using the losses to finally obtain the optimized scaling factor.

8. A block-by-block post-training quantization method for ViTs as claimed in claim 6 or 7, characterized in that: The loss for layer-wise optimization is: in, represents the mean square error loss, represents the mean operation, represents the square of the L2 norm, Indicates The output activation value of each module is Indicates The dequantization result of the output activation value of each module; The loss optimized by the stacked encoding module is: in, Indicates The total loss of the module is represents the balance factor, represents the loss of the attention-weighted metric, Indicates The attention features of each module, Indicates The dequantization result of the attention feature of the module, |·| represents the absolute value operation, represents the Hadamard product.

9. A block-by-block post-training quantization system for ViTs, characterized in that: The method is used to execute a block-by-block post-training quantization method for ViTs according to any one of claims 1 to 8, comprising: an acquisition module, a calibration module and an optimization module connected in sequence, The acquisition module is used to acquire a full-precision Vision transformer pre-trained model, calibration data, and a quantizer; The calibration module is used to replace all modules in the full-precision Vision transformer pre-trained model with corresponding quantization modules to obtain a ViTs model to be quantized, wherein the quantization module includes a quantizer of an activation value and a weight, and the activation value adopts a uniform quantizer per image block; using the calibration data, the scaling factor and the initial value of the zero offset of each quantizer in the ViTs model to be quantized are calculated; The optimization module is used to iteratively optimize the scaling factor of the ViTs model to be quantized by using the calibration data and the full-precision Vision transformer pre-training model to obtain a quantized ViTs model.

Images