A method for processing weight outliers when post-quantizing a model

Abstract

The application provides a method for processing weight outliers during post-model quantization. w The critical value is max w The maximum value of full-precision data, and b is the bit width after quantization; when S w Is less than the critical value, that is, when S x Is the scale of the input feature value Feature, when the weight and Feature are quantized to 8 bits, the maximum value of W int Is 128, and the maximum value of X int Is 255; at this time, the output quantization result is determined by b i Mainly, the weight is quantized to 0, and S w Is set to 1, thereby avoiding the problem that the quantization result is incorrect due to the weight outliers. The application solves the problem that the final model precision is lost due to the overflow of the calculated quantization hyperparameters caused by the weight data outliers during the post-model quantization process, and provides a method for improving the model quantization precision.

Description

Technical Field

[0001] This invention relates to the field of neural network model quantization technology, and in particular to a method for handling weight outliers during model post-quantization. Background Technology

[0002] In recent years, with the rapid development of science and technology, the era of big data has arrived. Deep learning, using deep neural networks (DNNs) as models, has achieved remarkable results in many key areas of artificial intelligence, such as image recognition, reinforcement learning, and semantic analysis. Convolutional neural networks (CNNs), as a typical DNN structure, can effectively extract hidden features from images and accurately classify them, and have been widely used in image recognition and detection in recent years.

[0003] The MNN inference framework provides a set of tools that can quantize a full-precision model into a low-bit model, and offers two methods for weight quantization: MAX_ABS and ADMM.

[0004] However, existing technologies do not handle outliers in the weights well, leading to a loss of accuracy in the quantized model.

[0005] In addition, the commonly used terms in the prior art are as follows:

[0006] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that incorporates convolutional computations and has a deep structure.

[0007] Model quantization: Model quantization is a technique that converts floating-point calculations into low-ratio fixed-point calculations. It can effectively reduce the computational intensity, parameter size, and memory consumption of the model, but often brings a huge loss of accuracy.

[0008] Post-quantization refers to directly quantizing a fully trained, high-precision model into a low-bit model.

[0009] Outliers in weights: Weights whose maximum value for some channels is very small, such as 1e-9. ImageNet: Like a network, it has multiple nodes. Each node is equivalent to an item or subcategory. The structure of ImageNet is basically a pyramid: directory -> subdirectories -> image sets. Summary of the Invention

[0010] To address the aforementioned issues, the purpose of this application is to resolve the problem of overflow of calculated quantization hyperparameters due to abnormal weight data during the post-quantization process, which leads to a loss of final model accuracy, and to provide a method for improving model quantization accuracy.

[0011] Specifically, the present invention provides a method for handling outlier weights during post-quantization of a model. This method provides special handling for layers exhibiting outlier weights and offers a method for calculating S... w The critical value method, in which, max w The maximum value of full-precision data, where b is the bit width after quantization; when S w Less than this critical value, that is: At that time, S x Scale is the input feature value. When the weights and features are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255, at which point the output quantization result is mainly determined by b. i The decision was made to quantize the weights to 0 and set S... w Set it to 1 to avoid incorrect quantization results due to outliers in the weights.

[0012] The method further includes:

[0013] S1, Convolutional Neural Network Training:

[0014] Using ImageNet image classification data as the dataset and ResNet-50 as the network structure, we trained a full-precision model to obtain the relevant parameters during the model's inference process.

[0015] S2, Model Preprocessing:

[0016] For the topology of the network from Conv to BN to ReLU, the BatchNormal layer will be merged into the convolution beforehand by merging and normalizing BatchNormal, so that it becomes the topology of Conv to Bias layer to ReLU.

[0017] S3, Quantization Formula: For the obtained weight parameters, quantize them according to the formula shown below to obtain the low-bit weights:

[0018]

[0019] Variable description: W f For full-precision data, W int For the quantized data, max w The maximum value of the full-precision data, where b is the bit width after quantization; S w The calculation is shown in the formula above, and it is the quantification factor of the weight;

[0020] S4, Calculation of quantization hyperparameters:

[0021] Suppose the quantization calculation for the i-th layer is as follows:

[0022]

[0023]

[0024] Where δ i Q is the activation function. A Q is the quantization formula for the feature map. w S is the formula for quantifying the weights. BN The scaling factor α is generated when BatchNormal is merged into the convolution. In this application, this value is incorporated into the weights, so S BN It equals 1.

[0025] The parameters in the above formula meet the following conditions:

[0026] 1) Represented using fixed-point numbers scaled from floating-point scalars. w int It is a fixed-point number represented by an integer;

[0027] 2) Represented using fixed-point numbers scaled from floating-point scalars. x int It is a fixed-point number represented by an integer;

[0028] 3) δ i It is a monotonic function;

[0029] Therefore, the final calculation It can be obtained through fixed-point number operations, that is:

[0030]

[0031]

[0032] If the weights of the network obtained by solving for the maximum value appear... w If the value is very small, then it makes The result overflow cannot be stored using a 32-bit integer (int32). As shown in the formula above, when max... w When the value of S is very small w The value of S will also be very small when the weights and features are quantized to 8 bits. w W int S x X int S BN The value of will also be very small, at which point the output quantization result is mainly determined by b. i Decide;

[0033] If you directly follow the formula To quantify b i This can lead to result overflow, resulting in incorrect quantization results for that layer; as can be seen from the quantization formula in step S3, when S... w W int S x X int S BN +b i When the value is less than 0.5, the quantized value is 0, where 0.5 is a fixed value.

[0034] S w W int S x X int S BN <0.5-b i

[0035] Where S x For the scale of the input feature, when the weights and feature are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255. At this point, the output quantization result is mainly determined by b. i Decide;

[0036] This allows all weights to be quantized to 0 and S to be... w Set it to 1 to avoid incorrect quantization results due to outliers in the weights.

[0037] In step S2, the specific merging formula is as follows:

[0038] Mean: μ, Variance: σ, Scaling factor: γ, Offset: β, A small number to prevent the denominator from being 0: ε;

[0039]

[0040]

[0041] After merging the BN layer and the convolutional layer:

[0042]

[0043] W merged =W×α

[0044] W merged =B×α+(β-μ×α).

[0045] In step S4, the network weights are obtained by solving the gradient descent method.

[0046] In step S4, the occurrence of max wA value less than or equal to 1e-8 will cause parameter overflow, and 1e-20 is preferred.

[0047] The method is based on equation S. w W int S x X int S BN +b i <0.5 to confirm W int and S w The value of .

[0048] Therefore, the advantage of this application is that it performs special processing on layers with outlier weights, reduces the loss of model accuracy after quantization, and improves the accuracy of the final model. Attached Figure Description

[0049] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0050] Figure 1 This is a schematic diagram of the network topology of the method involved in this application.

[0051] Figure 2 yes Figure 1 A schematic diagram of the topology after being merged into convolution.

[0052] Figure 3 This is a flowchart illustrating the method involved in this application. Detailed Implementation

[0053] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0054] This method is a way to handle outlier weights, aiming to overcome the defects in the existing technology and solve the problem of loss of accuracy in the final model due to overflow of calculated quantization hyperparameters caused by abnormal weight data.

[0055] To solve the above-mentioned technical problems, the technical solution created by this invention is implemented as follows:

[0056] 1) Training of Convolutional Neural Networks:

[0057] Using ImageNet image classification data as the dataset and ResNet-50 as the network structure, a full-precision model is trained to obtain the relevant parameters during the model's inference process.

[0058] 2) Model preprocessing:

[0059] For the network Figure 1The topology will be pre-processed by merging BatchNormal operations into the convolutions, resulting in something like... Figure 2 The displayed topology and specific merging formula are shown below.

[0060] Mean: μ, Variance: σ, Scaling factor: γ, Offset: β, A small number to prevent the denominator from being 0: ε;

[0061]

[0062]

[0063] After merging the BN layer and the convolutional layer:

[0064]

[0065] W merged =W×α

[0066] W merged =B×α+(β-μ×α).

[0067] 3) Quantization formula: For the obtained weight parameters, quantize them according to the formula shown below to obtain the low-bit weights;

[0068]

[0069] Variable description: W f For full-precision data, W int For the quantized data, max w The maximum value of the full-precision data, where b is the bit width after quantization.

[0070] 4) Calculation of quantization hyperparameters:

[0071] Suppose the quantization calculation for the i-th layer is as follows:

[0072]

[0073]

[0074] Where δ i Q is the activation function. A The quantization formula for the feature map, Q w The parameters in the above formula for quantifying weights must satisfy the following conditions:

[0075] 1. It can be represented using a fixed-point number scaled from a floating-point scalar, i.e. w int It is a fixed-point number represented by an integer;

[0076] 2. It can be represented using a fixed-point number scaled from a floating-point scalar, i.e. x int It is a fixed-point number represented by an integer;

[0077] 3. δ i It is a monotonic function;

[0078] Therefore, the final calculation It can be obtained through fixed-point number operations, that is:

[0079]

[0080] Since the network weights are obtained using the gradient descent method, a maximum value may occur. w The value is very small (e.g., 1e-20), thus making The result overflow cannot be stored using int32. As shown in the above formula, when max... w When the value of S is very small w The value of S will also be very small when the weights and features are quantized to 8 bits. w W int S x X int S BN The value of will also be very small, at which point the output quantization result is mainly determined by b. i Decision. If we directly follow the formula... To quantify b i This can lead to result overflow, causing the quantization result of that layer to be incorrect. From the quantization formula in step S3, it can be seen that when S... w W int S x X int S BN +b i When the value is less than 0.5, the quantized value is 0, that is...

[0081] S w W int S x X int S BN <0.5-b i Where S x For the scale of the input feature, when the weights and feature are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255. At this point, the output quantization result is mainly determined by b. i If the decision is made, then all weights can be quantized to 0 and S can be set to 0. w Set it to 1 to avoid incorrect quantization results due to outliers in the weights.

[0082] The gradient descent method is an optimization algorithm. The calculation process of gradient descent is to find the minimum value along the direction of gradient descent.

[0083] In summary, such as Figure 3 As shown, this method further includes:

[0084] S1, Convolutional Neural Network Training:

[0085] Using ImageNet image classification data as the dataset and ResNet-50 as the network structure, we trained a full-precision model to obtain the relevant parameters during the model's inference process.

[0086] S2, Model Preprocessing:

[0087] For the topology of the network from Conv to BN to ReLU, the BatchNormal layer will be merged into the convolution beforehand by merging and normalizing BatchNormal, so that it becomes the topology of Conv to Bias layer to ReLU.

[0088] S3, Quantization Formula: For the obtained weight parameters, quantize them according to the formula shown below to obtain the low-bit weights:

[0089]

[0090] Variable description: W f For full-precision data, W int The data is quantized, and the maximum value is [value]. w The maximum value of the full-precision data, where b is the bit width after quantization; S w It is the quantification factor of the weight;

[0091] S4, Calculation of quantization hyperparameters:

[0092] When S w Less than this critical value, that is: At that time, S x Scale is the input feature value. When the weights and features are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255, at which point the output quantization result is determined by b. i The decision is primary, the weight is quantified to 0, and S is... w Set it to 1 to avoid incorrect quantization results due to outliers in the weights.

[0093] The method in this application is to perform special processing on layers with outlier weights, thereby reducing the loss of model accuracy after quantization and improving the accuracy of the final model.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for handling outlier weights during post-quantization of a model, characterized in that, The method provides special handling for layers with outlier weights, namely, providing a calculation method for S. w The critical value method, in which, max w where b is the maximum value of the full-precision data and b is the bit width after quantization; when S w Less than this critical value, that is: At that time, S x Scale is the input feature value. When the weights and features are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255, at which point the output quantization result is determined by b. i The decision is primary, the weight is quantified to 0, and S is... w Set it to 1 to avoid incorrect quantization results due to outlier weights; The method further includes: S1, Convolutional Neural Network Training: Using ImageNet image classification data as the dataset and ResNet-50 as the network structure, we trained a full-precision model to obtain the relevant parameters during the model's inference process. S2, Model Preprocessing: For the topology of the network from Conv to BN to ReLU, the BatchNormal layer will be merged into the convolution beforehand by merging and normalizing BatchNormal, so that it becomes the topology of Conv to Bias layer to ReLU. S3, Quantization Formula: For the obtained weight parameters, quantize them according to the formula shown below to obtain the low-bit weights: Variable description: W f For full-precision data, W int The data is quantized, and the maximum value is [value]. w The maximum value of the full-precision data, where b is the bit width after quantization; S w It is the quantification factor of the weight; S4, Calculation of quantization hyperparameters: Suppose the quantization calculation for the i-th layer is as follows: Where δ i Q is the activation function. A Q is the quantization formula for the feature map. w S is the formula for quantifying the weights. BN The scaling factor α is generated when BatchNormal is merged into the convolution. In this application, this value is incorporated into the weights, so S BN It equals 1; Calculate the final It can be obtained through fixed-point number operations, that is: If the weights of the network obtained by solving for the maximum value appear... w If the value is very small, then it makes The result overflow cannot be stored using a 32-bit integer (int32). As shown in the formula above, when max... w When the value of S is very small w The value of S will also be very small when the weights and features are quantized to 8 bits. w W int S x X int S BN The value will also be a very small value, at which point the output quantization result is determined by b. i The decision is primary; If you directly follow the formula To quantify b i This can lead to result overflow, resulting in incorrect quantization results for that layer; as can be seen from the quantization formula in step S3, when S w W int S x X int S BN +b i When the value is less than 0.5, the quantized value is 0, that is... S w W int S x X int S BN <0.5-b i Where S x For the scale of the input feature, when the weights and feature are quantized to 8 bits, W int The maximum value is 128, X int The maximum value is 255. At this point, the output quantization result is determined by b. i The decision is primary; This allows all weights to be quantized to 0 and S to be... w Set it to 1 to avoid incorrect quantization results due to outliers in the weights.

2. The method for processing outlier weights during post-quantization of a model according to claim 1, characterized in that, In step S2, the specific merging formula is as follows: Mean: μ, Variance: σ, Scaling factor: γ, Offset: β, A small number to prevent the denominator from being 0: ε; After merging the BN layer and the convolutional layer: IN merged =W×α B merged = B×α+(β-μ×α).

3. The method for processing outlier weights during post-quantization of a model according to claim 1, characterized in that, In step S4, the occurrence of max w A value less than or equal to 1e-8 will cause parameter overflow.

4. The method for processing outlier weights during post-quantization of a model according to claim 3, characterized in that, The max w The value is 1e-20.

5. The method for processing outlier weights during post-quantization of a model according to claim 1, characterized in that, In step S4, the parameters in the formula meet the following conditions: 1) Represented using fixed-point numbers scaled from floating-point scalars. w int It is a fixed-point number represented by an integer; 2) Represented using fixed-point numbers scaled from floating-point scalars. x int It is a fixed-point number represented by an integer; 3) δ i It is a monotonic function.

6. The method for processing outlier weights during post-quantization of a model according to claim 1, characterized in that, In step S4, the network weights are obtained by solving the gradient descent method.

7. The method for processing outlier weights during post-quantization of a model according to claim 1, characterized in that, Through the equation S w W int S x X int S BN +b i <0.5 to confirm W int and S w The value of .

Images