Linear asymmetric quantization method based on bias and scale

Abstract

The invention provides a linear asymmetric quantization method based on bias and scale. The method is optimization of conversion from floating point calculation to fixed point calculation in model quantization. According to the method, model quantification takes loss of reasoning precision as the cost, floating-point type parameters (weight and input feature) of continuous values or discrete values in a network are linearly mapped into discrete values of fixed-point approximation (int8 / uint8), original float32 format data are replaced, meanwhile, input and output are kept in a floating-point type, and the method has the advantages of being simple in structure, low in cost and high in reliability. Therefore, the purposes of reducing the model size, reducing the model memory consumption, increasing the model reasoning speed and the like are achieved. The method can be used for realizing model quantification of tasks such as detection and classification in the field of artificial intelligence.

Description

Technical Field

[0001] The present invention belongs to the technical field of model compression, and particularly relates to a linear asymmetric quantization method based on bias and scale. Background Art

[0002] Because the computing power and memory of embedded devices are limited, deep learning models need to be compressed before they can be better deployed on embedded devices. To a certain extent, the deeper the network and the more parameters there are, the more complex the model will be, but the better its final effect will be. And model compression algorithms aim to convert a large and complex pre-trained model into a streamlined small model.

[0003] Model quantization is a popular deep learning optimization method in which model data (including network parameters and activations) is converted from a floating-point representation to a lower-precision representation, usually using 8-bit integers. This has several benefits: ① When processing 8-bit integer data, NVIDIA GPUs use faster and lower-cost 8-bit tensor cores to compute convolution and matrix multiplication operations. This results in more computational throughput, which is particularly effective on computationally limited layers. ② Moving data from memory to the computational unit (the streaming multiprocessors in NVIDIA GPUs) takes time and energy, and also generates heat. Reducing the precision of activation and parameter data from 32-bit floating-point values to 8-bit integers can result in a 4-fold reduction in data, thus achieving the purpose of saving power and reducing the heat generated. ③ Since some layers have bandwidth limitations (limited memory), this means that their implementation spends most of the time reading and writing data, so reducing their computational time does not reduce their overall running time. Bandwidth-limited layers benefit the most from reduced bandwidth requirements. ④ Reducing memory occupancy means that the model requires less storage space, smaller parameter updates, higher cache utilization, etc.

[0004] However, current models all run on four types of chips, namely CPUs, GPUs, FPGAs, and ASICs. The computing power of the chips is limited; for the chips on edge devices, there are many limitations in terms of storage, memory, power consumption, and latency, and inference efficiency is particularly important.

[0005] In addition, common terms in the prior art include:

[0006] Model quantization: Quantization is a method for compressing and accelerating the inference of neural network models. This method maps the weights or activation functions of deep learning models from 32-bit floating-point numbers to data representations with a lower bit depth.

[0007] Linear quantization: A common model quantization method in industry. It uses uniformly distributed clustering centers, and there is a simple linear transformation relationship between the original floating-point data and the quantized fixed-point data. At the same time, convolution and fully connected layers themselves are also simple linear calculations. Therefore, in linear quantization, the quantized data can be directly used for direct calculation, which can not only compress the size of the model file but also bring a significant speed improvement. According to whether the offset parameter in quantization is zero, linear quantization can be divided into two categories, namely symmetric and asymmetric quantization. Among them, the asymmetric quantization method described in this application can generally handle the situation of uneven data distribution better. Summary of the Invention

[0008] To solve the above problems, the purpose of this application is as follows: In this method, model quantization sacrifices inference accuracy and linearly maps the floating-point parameters (weights and inputs) with continuous or discrete values in the network to discrete values approximated by fixed-point (int8 / uint8), replacing the original float32 format data, while keeping the input and output as floating-point types, so as to achieve the goals of reducing the model size, reducing the model memory consumption, and accelerating the model inference speed. This method can be used for model quantization implementation in tasks such as detection and classification in the field of artificial intelligence.

[0009] Specifically, the present invention provides a linear asymmetric quantization method based on bias and scale, and the method includes:

[0010] For asymmetric quantization, a mapping formula is determined by a scaling factor S and a zero-point offset Z to map the input floating-point data to fixed-point data within the range of [0, 255]. Let min(R) represent the minimum value of the input floating-point data, and max(R) represent the maximum value of the input floating-point data. Assuming the fixed-point Q is represented by the unsigned integer value UINT8, at this time Z≠0.

[0011] S1, Optimization of the conversion from floating-point operations to fixed-point calculations in model quantization, with X f representing the original FP32 data and X q representing the quantized UINT8 data, the conversion is shown in formula (1):

[0012]

[0013] At this time, the quantization parameters S and Z can be expressed as:

[0014]

[0015]

[0016] Among them, n represents the quantization bit number. At this time, substituting formula (2) and formula (3) into formula (1) can obtain:

[0017]

[0018] Formula (4) is the formula for converting floating-point data to fixed-point data;

[0019] The convolution formula is expressed as:

[0020]

[0021] where is the result of convolving the input data x with the weight w, b is the bias, and y is the output result;

[0022] According to Formula (4) and Formula (5), a fixed-point convolution calculation formula can be obtained to implement quantized fixed-point calculation: The BN calculation formula is expressed as:

[0023]

[0024] where X conv is the convolution result, mean is the new mean calculated in a moving average manner, var is the new variance calculated in a moving average manner, gama is the scaling factor, beta is the translation factor, and eps is a value close to 0 used to prevent the denominator from being 0;

[0025] In S2, keep the input and output as floating-point types. The essence of the BN is to normalize first and then scale and translate. Combining the formula in step S1, the following derivation can be obtained:

[0026]

[0027] where Conv_bn is the floating-point output after the input data x passes through convolution and BatchNormal, and conv is the floating-point result after the input data x passes through convolution The weight Weight dequantization formula is w = w int × w_scale, the activation Feature quantization formula is x = (x int - in_offset) × in_scale, and assume Scale = bn_scale × w_scale × in_scale, is the operation of convolving the quantized input value and weight value, where w_scale is the scaling factor for quantizing the floating-point weight data w, and in_scale is the scaling factor for quantizing the floating-point input data x.

[0028] In the formula (2), n = 8 in INT8.

[0029] The eps is a value close to 0, used to prevent the denominator from being 0, and is represented by 1×e -05 It is represented as

[0030] The method is applied to the process of data flow passing through the convolutional layer and the batch normalization layer during the inference process of the neural network model, converting the original floating-point calculation into a fixed-point integer calculation, and simplifying the calculation of batch normalization to the operation of adding Bias to the convolution result and then multiplying by Scale

[0031] For the activation feature parameter with relatively concentrated data distribution, the asymmetric quantization of this method is used

[0032] The method is applicable to the model quantization implementation of detection and classification tasks in the field of artificial intelligence

[0033] Therefore, the advantages of this application are as follows: The asymmetric quantization method described in this application converts the floating-point calculation in Conv_BN into a fixed-point calculation, reduces the memory occupation of data in the model, and speeds up the model inference operation efficiency, thereby significantly improving the inference speed of the algorithm and reducing the model memory occupation, making it easier for the AI model to be deployed on the chip Description of the Drawings

[0034] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not constitute a limitation to the present invention

[0035] Figure 1 It is a schematic diagram of the asymmetric mapping from FP32 to UINT8

[0036] Figure 2 It is a schematic diagram of the process flow of this method

[0037] Figure 3 It is a schematic diagram of a part of the c++ calculation code for implementing the calculation process of this method Detailed Embodiments

[0038] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings

[0039] The present invention belongs to an optimized implementation of a model quantization method in the field of model compression. The main content of this application is the derivation of the quantization formula for Conv-BN and the implementation of Bias and Scale quantization (calculated by channels). Generally speaking, symmetric quantization is used for the weight parameter with relatively symmetric data distribution; asymmetric quantization is used for the activation feature parameter with relatively concentrated data distribution

[0040] For asymmetric quantization, such asFigure 1 As shown in the figure, the upper horizontal line in the figure represents the floating-point interval before quantization, and the lower horizontal line in the figure represents the unsigned INT8 interval (i.e., UINT8) after quantization. That is, a mapping formula is determined through a scaling factor (denoted as S) and a zero-point offset (denoted as Z) to map the input floating-point data to fixed-point data within the range of [0, 255]. min(R) in the figure represents the minimum value of the input floating-point data, and max(R) represents the maximum value of the input floating-point data. Assume that the fixed-point Q is represented by the unsigned integer value UINT8, and at this time Z≠0. Specifically, the steps of this method are as Figure 2 shown:

[0041] S1. Optimization of the conversion from floating-point operations to fixed-point calculations in model quantization. The conversion between the original FP32 data (denoted as Xf) and the quantized UINT8 data (denoted as X q is shown in formula (1):

[0042]

[0043] At this time, the quantization parameters S and Z can be expressed as:

[0044]

[0045] [[ID=2,2]]

[0046] where n represents the quantization bit number, and in INT8, n = 8;

[0047] The program code for calculating the quantization parameter in_scale (scaling factor) of the floating-point input data x according to formula (2) is as follows:

[0048] float input_max,input_min;

[0049] int in_bit;

[0050] float in_scale=(input_max-input_min) / (pow(2,in_bit)-1);

[0051] The program code for calculating the quantization parameter w_scale (scaling factor) of the floating-point weight data w according to formula (2) is as follows:

[0052] floatweight_max,weight_min;

[0053] intweight_bit;

[0054] float w_scale = [abs(weight_min) > abs(weight_max)? abs(weight_min) :

[0055] abs(weight_max)] / pow(2, weight_bit) - 1;

[0056] The program code for calculating the quantization parameter in_offset (zero - point offset) of the floating - point input data x according to formula (3) is shown as follows:

[0057] float in_offset = std::rint(-input_min / in_scale);

[0058] At this time, substituting formula (2) and formula (3) into formula (1) can obtain:

[0059]

[0060] Formula (4) is the formula for converting floating - point data to fixed - point data;

[0061] Convolution formula:

[0062]

[0063] Among them, is the result of convolving the input data x and the weight w, b is the bias, and y is the output result.

[0064] According to formula (4) and formula (5), the fixed - point convolution calculation formula can be obtained to implement the quantization fixed - point calculation: The BN calculation formula is expressed as:

[0065]

[0066] Among them, X conv is the convolution result, mean is the new mean calculated in the way of moving average, var is the new variance calculated in the way of moving average, gama is the scaling factor, beta is the translation factor, eps is a value close to 0, used to prevent the denominator from being 0 (usually expressed as 1×e -05 ).

[0067] S2, keeping the input and output as floating - point types, the essence of BN is actually to normalize first and then scale and translate; Combining the above formula can obtain the following derivation:

[0068]

[0069] Among them, Conv_bn is the floating-point output after the input data x passes through convolution and BatchNormal, and conv is the floating-point result after the input data x passes through convolution. The weight quantization formula for Weight is w = w int × w_scale, and the activation quantization formula for Feature is x = (x int - in_offset) × in_scale, and it is assumed that Scale = bn_scale × w-scale × in_scal. is the operation of performing convolution using the quantized input values and weight values, where w_scale is the scaling factor for quantizing the floating-point weight data w, and in_scale is the scaling factor for quantizing the floating-point input data x.

[0070] The program code representation of formula (7) is as follows:

[0071] float gamma_value, var_value, epsilon;

[0072] float bn_scale = gamma_value / (std::sqrt(var_value + epsilon));

[0073] The calculation code for Scale in formula (7) is:

[0074] float Scale = bn_scale * in_scale * w_scale;

[0075] The calculation code for Bias in formula (7) is:

[0076] float bias_value, mean_value, beta_value;

[0077] float Bias = bn_scale * (bias_value - mean_value) + beta_value;

[0078] The calculation code for calibration in formula (7) is:

[0079]

[0080] The calculation code for Conv_bn in formula (7) is: where Conv_bn = Scale * (Conv + Bias / Scale - calibration).

[0081] Part of the C++ code for the above calculation process is as follows Figure 3 shown below

[0082] The code section converts the original floating-point calculations into fixed-point integer calculations when the data stream passes through the convolutional layer and the batch normalization layer during the inference process of the neural network model, and simplifies the calculation of batch normalization to adding Bias to the convolutional result and then multiplying by Scale. This will further compress the model size, reduce the data calculation amount on the chip side, and accelerate the model inference speed

[0083] In addition, in the alternative solutions used in combination with this method, there are also

[0084] Similar solutions include the linear symmetric quantization method. Compared with symmetric quantization, asymmetric quantization does not need to follow the mapping rule of unchanged zero point, has a better dynamic mapping range, and when facing some special situations, such as for the activation values (all non-negative values) after Relu, when using symmetric quantization, it is necessary to carefully consider using signed quantization or unsigned quantization methods, while using asymmetric quantization does not need to be considered; on the other hand, asymmetric quantization does require a greater computational cost compared to symmetric quantization. Therefore, on the premise of hardware support (such as Junzheng T40, T41 chips, etc.), using asymmetric quantization for the activation value Feature and symmetric quantization for the weight value Weight during quantization may be a more appropriate quantization scheme

[0085] The above are only the preferred embodiments of the present invention, and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention

Claims

1. A linear asymmetric quantization method based on bias and scale, characterized in that, The method includes: For asymmetric quantization, a mapping formula is determined by a scaling factor scale denoted as S and a zero point offset zero_point denoted as Z to map the input floating-point data to fixed-point data within the range of [0, 255]. Let min(R) represent the minimum value of the input floating-point data, and max(R) represent the maximum value of the input floating-point data. Assume that the fixed point Q is represented by an unsigned integer value UINT8, and at this time Z≠0; S1. Optimization of the conversion from floating-point operations to fixed-point calculations in model quantization, taking X f The conversion between the original FP32 data represented by X q and the quantized UINT8 data represented by X is shown in Equation (1): At this time, the quantization parameters S and Z can be expressed as: Among them, n represents the quantization bit number. At this time, substituting formula (2) and formula (3) into formula (1) can obtain: Formula (4) is the formula for converting floating-point data to fixed-point data; The convolution formula is expressed as: Formula (5) is the floating-point convolution calculation formula, where is the result of convolving the input data x with the weight w, b is the bias, and y is the output result; According to formula (4) and formula (5), the fixed-point convolution calculation formula can be obtained to realize the fixed-point calculation of quantization: The BN calculation formula is expressed as: where X conv is the convolution result, mean is the new mean calculated in the way of moving average, var is the new variance calculated in the way of moving average, gama is the scaling factor, beta is the translation factor, and eps is a value close to 0 used to prevent the denominator from being 0; S2, keeping the input and output as floating-point types. The essence of the BN is to normalize first and then scale and translate. Combining with the formula in step S1, the following derivation can be obtained: Among them, Conv_bn is the floating-point output after the input data x passes through convolution and BatchNormal, and conv is the floating-point result after the input data x passes through convolution. The weight de-quantization formula is w = w int × w_scale, and the activation feature quantization formula is x = (x int - in_offset) × in_scale, and it is assumed that Scale = bn_scale × w_scale × in_scale. is the operation of performing convolution using the quantized input value and weight value, where w_scale is the scaling factor for quantizing the floating-point weight data w, and in_scale is the scaling factor for quantizing the floating-point input data x.

2. The linear asymmetric quantization method based on bias and scale according to claim 1, characterized in that, In the formula (2), n = 8 in INT8.

3. A linear asymmetric quantization method based on bias and scale according to claim 1, characterized in that, The eps is a value close to 0, used to prevent the denominator from being 0, expressed as 1×e -05 represent.

4. A linear asymmetric quantization method based on bias and scale according to claim 1, wherein The method is applied to the process of neural network model inference. When the data stream passes through the convolutional layer and the batch normalization layer, the original floating-point calculation is converted into fixed-point integer calculation, and the calculation of batch normalization is simplified to the operation of adding Bias to the convolution result and then multiplying by Scale.

5. A linear asymmetric quantization method based on bias and scale according to claim 1, characterized in that The asymmetric quantization of this method is used for the activation feature feature parameters with relatively concentrated data distribution.

6. A linear asymmetric quantization method based on bias and scale according to claim 1, characterized in that The method is applicable to the model quantization implementation of detection and classification tasks in the field of artificial intelligence.

Images