A method for quickly combining various quantization algorithms and MNNs

Abstract

The application provides a method for quickly integrating a quantization method into an MNN, and is particularly useful for a quantization algorithm with a complex quantization process. The process of obtaining quantization parameters S and Z still follows the original flow. Then, the ONNX is simplified offline, unnecessary nodes are removed, corresponding parameters are extracted, and the parameters are integrated into the MNN. The modification is simple and convenient. The application is a practical, simple and fast method for quickly combining a model quantized by other methods with an MNN.

Description

Technical Field

[0001] This invention belongs to the field of neural network technology, and specifically relates to a method for rapidly combining various quantization algorithms with MNN. Background Technology

[0002] Convolutional neural networks (CNNs) boast high accuracy, even surpassing human precision in some tasks such as face recognition and image classification. However, their drawbacks are also significant, including a large number of parameters, high computational cost, and high memory consumption. Model quantization can alleviate these problems, offering potential advantages such as parameter compression, speed improvement, and reduced memory usage. The word "potential" is in quotes because achieving all three simultaneously is challenging, with numerous limitations and prerequisites in practical applications. Furthermore, since model quantization is an approximation algorithm, accuracy loss is a serious issue, and most research focuses on this problem.

[0003] Currently, with the continuous improvement of mobile phone computing power and the rapid development of deep learning, especially the increasing maturity of small network models, inference and prediction that were originally performed in the cloud can be transferred to the terminal. Edge intelligence refers to the deployment and operation of AI algorithms on the device side. Compared with server-side intelligence, edge intelligence has advantages such as low latency, protection of data privacy, and saving cloud resources.

[0004] Furthermore, MNN (Mobile Neural Network) is a lightweight deep neural network inference engine that loads deep neural network models on the device side for inference and prediction. Currently, MNN is used in over 20 apps, including Alibaba's Taobao Mobile, Tmall Mobile, and Youku, covering scenarios such as live streaming, short videos, search recommendations, product image search, interactive marketing, benefits distribution, and security risk control. It also has several applications in IoT and other scenarios.

[0005] AdaRound, AdaQuant, and BRECQ are currently cutting-edge strategies in post-training quantization (PTQ) research. They all evolve along the direction of "quantization parameter optimization with unchanged weights," meaning they all belong to optimization-based methods. When the quantization accuracy meets expectations, it is necessary to consider combining them with inference engines (such as MNN, TensorRT, TVM, etc.) to obtain a real inference acceleration effect. If the quantization method is relatively simple, the original MNN algorithm can be replaced. However, if the quantization method is more complex, such as requiring fine-tuning with small data involving backpropagation of parameters, and given that MNN is implemented in C++, merging them becomes very troublesome and prone to errors.

[0006] In addition, the following are commonly used terms in the prior art:

[0007] 1. Model Quantization: Quantization refers to the process of approximating the continuous values ​​of a signal into a finite number of discrete values. It can be understood as a method of information compression. In computer systems, this concept is generally represented by "low-bit" quantization. Some also call quantization "fixed-point quantization," but strictly speaking, the range it represents is reduced. Fixed-point quantization specifically refers to linear quantization with a scale of powers of 2, which is a more practical quantization method.

[0008] 2. Quantization-Aware-Training (QAT): This method simulates quantization behavior during training, using floating-point numbers to store fixed-point parameters. During inference, these fixed-point parameters are directly used. Advantages: Quantization constraints are considered during training, resulting in relatively better accuracy; it is more conducive to the implementation of extremely low-bit quantization (such as INT4 or INT8 / INT4 hybrid). Disadvantages: There are training costs, and maintenance and upgrades are relatively difficult; accuracy may not be good, and it also depends on strategy selection and hyperparameter tuning.

[0009] 3. Post-quantization-training (PQT): This involves selecting appropriate quantization and calibration operations for the pre-trained network to minimize the quantization loss. This process is further divided into two types:

[0010] (1) Calibration data is required. These data are mainly used to obtain quantization parameters, so they do not need to be labeled. Generally, a hundred or so images are sufficient.

[0011] (2) No dataset is required at all. This is suitable for scenarios where the training environment and data are completely unavailable. Advantages: Simple and easy to use, and the quantization effect can be seen quickly; Disadvantages: Since quantization constraints are not considered during model training, quantization errors or accuracy loss may not be completely eliminated.

[0012] 4. The role of quantization: Quantization transforms the main operators (convolution) in the network from floating-point calculations to low-precision Int8 calculations, reducing model size and improving performance.

[0013] 5. ONNX (Open Neural Exchange) format is an open file format designed for machine learning to store trained models. It's a standard for representing deep learning models, enabling model transfer between different frameworks. Summary of the Invention

[0014] To address the aforementioned issues, the purpose of this application is to provide a practical, simple, and rapid method for quickly combining models quantized by other methods with MNNs.

[0015] Specifically, the present invention provides a method for rapidly combining various quantization algorithms with MNNs, the method comprising:

[0016] S1, Analysis of linear quantization methods:

[0017] set up:

[0018] Quantize X_int = round(X*S) + Z

[0019] X_Q = clamp(0, N-1, X_int)

[0020] De-quantize X_float=(X_Q-Z) / S

[0021] Where X is the original value of the float32, S is the scaling factor of the float32, Z is the offset of the float32, and N = 2. n -1, where n represents the quantization bit width, and N is calculated from bits; X_int is the quantized value compared to X_float, X_Q is the quantized value, and X_int becomes X_Q after boundary processing. In most cases, X_int and X_Q are equal. X_float is the dequantized value, which is assumed to be already quantized, that is, the appropriate S and Z have been obtained.

[0022] S2, assuming a linearly quantized convolutional layer, a series of nodes above node Conv quantize the weights (Weight), followed by the quantization of the features. The goal is to obtain only the quantization parameters: the scaling factor S_W and zeros Z_W for the weights, and the scaling factor S_F and zeros Z_F for the features; further including:

[0023] (1) The original weight is first divided by the S value, which is expressed as: Div: W' = W / S;

[0024] (2), round() is the rounding function, expressed as: Aten:W_r=round(W');

[0025] (3), this step is equivalent to calculating the difference between the integer and the floating point before rounding, expressed as: Sub: d = W_r - W';

[0026] (4), the result of this step is actually the same as the result after rounding in (2), which is represented as Add: W_int=W'+d;

[0027] (5) Adding the zero point value, it is expressed as: Add: W_int=W_int+Z_W

[0028] (6) This operation is to prevent out-of-bounds errors. A phase is performed to limit the value to the quantized range of 0 to 2. bit -1; is represented as: Clip: W_Q = clamp(W_int, 0, N), where N = 2 bit -1;

[0029] (7) is equivalent to subtracting zero from the previously quantized value, which is expressed as: Sub: W_Q = W_Q - Z_W;

[0030] (8) Multiply by the previous scale value. These two steps are equivalent to dequantization, which can be expressed as: Mul: W_float=W_Q*S;

[0031] Therefore, steps (1)-(6) above are performing quantization operations, corresponding to X_Q=clamp(0,N-1,round(X*S)+Z); (7)-(8) are performing dequantization operations, corresponding to X_float=(X_Q-Z) / S; therefore, the quantized S value can be taken out from Div in step (1), and the Z value is taken out from the second Sub, i.e., step (7). Since this is the quantization of Weight, S_W and Z_W are used to represent it.

[0032] S3, from the analysis of the quantization method in S1 and the nodes in S2, we can see that the value of S can be obtained from the division Div. The input[0] of the Div before node Conv corresponds to the weight of Conv, and the input[1] corresponds to the S_W of Conv. The input[1] of the Div after Conv corresponds to the scaling factor S_F of Conv. Among them, the division Div has two inputs, input[0] represents the first input and input[1] represents the second input. The value of Z is obtained from the subtraction Sub. There are two Subs before and after Conv, but only the second Sub of each is the real zero point. Z can be obtained by judging the length of the node's attribute. When the dimension of the node's attribute value node.attribute.t.dims has a value, it is the zero point Z_W, otherwise it is Z_F. The single value is dim. s is 0, and multiple values, such as weight, have multiple values ​​with dimensions [32, 1, 1, 1], where 32 is the number of channels for the current layer's weight; S4, finally yielding the simplified ONNX. From the Attributes, we can see that in addition to the original Conv attributes, the necessary quantization parameters S_W, S_F, Z_W, and Z_F have been added. When converting to *.mnn format, these parameters will be modified accordingly. During forward propagation, the parameters will be read and extracted for quantization to replace the existing quantization method. That is, because quantization in MNN is completed during conversion, but the current ONNX has already been quantized (i.e., S and Z are already obtained), there is no need to calculate S and Z separately; only the input data needs to be quantized. In PyTorch, forward represents a forward propagation, indicating the sequential computation steps for building network layers.

[0033] In step S2, the convolutional layer of the linear quantization has already had BN merged before quantization.

[0034] In step S3, the input[0] of the Div before the convolution Conv corresponds to the weight of Conv, and the input[1] corresponds to the S_W of Conv, which is divided into channels, with 'channels' values.

[0035] In step S3, there are two Subs before and after Conv, and the first Sub of each is the calculation of the difference between integer and floating-point values.

[0036] In step S3, when the dimension of the node's attribute value, node.attribute.t.dims, has a value, it is zero point Z_W because Weight is divided into channels and has multiple values; otherwise, it is Z_F, that is, Feature is not divided into channels and has a single value.

[0037] In step S3, in order to ensure the continuity between all network nodes after removing useless nodes, the output node of Conv is changed to the output node of the last multiplication Mul of the quantized feature, so that the Conv layer is naturally connected to the next layer.

[0038] Therefore, the advantage of this application is:

[0039] This application proposes a rapid method for integrating post-quantization methods into a Multi-Neural Network (MNN), which is particularly useful for quantization algorithms with complex quantization processes. The process of obtaining quantization parameters S and Z still follows the original workflow. Afterwards, ONNX is simplified offline by removing unnecessary nodes, extracting the corresponding parameters, and then integrating them into the MNN. Modification is simple and convenient. For other quantization methods, especially those requiring small batches of correction data for backpropagation to obtain S, Z, or other quantization parameters, this method allows for the modification of ONNX nodes to meet the requirements of the MNN. During modification, only the parameter extraction correspondence and the connection relationships between node inputs and outputs need to be considered. Attached Figure Description

[0040] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0041] Figure 1 This is a schematic diagram of a convolutional layer for linear quantization according to the present invention.

[0042] Figure 2 This is a simplified ONNX diagram of the present invention.

[0043] Figure 3 This is a partial code diagram of step S2 of the present invention.

[0044] Figure 4 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0045] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0046] like Figure 4 As shown, this invention relates to a method for rapidly combining various quantization algorithms with MNNs, the method comprising:

[0047] S1, Analysis of linear quantization methods:

[0048] set up:

[0049] Quantize X_int = round(X*S) + Z

[0050] X_Q = clamp(0, N-1, X_int)

[0051] De-quantize X_float=(X_Q-Z) / S

[0052] Where X is the original value of the float32, S is the scaling factor of the float32, Z is the offset of the float32, and N = 2. n -1, where n represents the quantization bit width, and N is calculated from bits; X_int is the quantized value compared to X_float, X_Q is the quantized value, and X_float is the dequantized value. Here, it is assumed that quantization has been completed, that is, the appropriate S and Z have been obtained.

[0053] S2, assuming a linearly quantized convolutional layer, a series of nodes above node Conv quantize the weights (Weight), followed by the quantization of the features. The goal is to obtain only the quantization parameters, namely the scaling factor S_W and zeros Z_W for the weights, and the scaling factor S_F and zeros Z_F for the features; further including:

[0054] (1) The original weight is first divided by the S value, which is expressed as: Div: W' = W / S;

[0055] (2), round() is the rounding function, expressed as: Aten:W_r=round(W');

[0056] (3) This step is equivalent to calculating the difference between the integer and the floating point before rounding, which is expressed as: Sub: d = W_r - W';

[0057] (4) The result of this step is actually the same as the result after rounding in (2), which is represented as Add: W_int=W'+d; the reason why we need to do more steps (2) and (3) is as follows: Figure 3 The source code shows that the comments explain the implementation of a "pass-through estimator" for rounding operations. This approach is called a "pass-through estimator" (STE), which simply passes the gradient to these functions as is. In deep learning, parameters are generally learned continuously so that gradients can be updated during backpropagation. However, sometimes parameters are discrete, making backpropagation impossible, such as with `x.round()`. Therefore, the implementation uses a subtraction-then-add approach, preserving the original logic while simultaneously calculating the gradient (the tensor is detached, equivalent to a constant with a gradient of 0, while the gradient of `X` is 1), achieving two goals at once.

[0058] (5) Add the zero point value, which is expressed as: Add: W_int=W_int+Z_W;

[0059] (6) This operation is to prevent out-of-bounds errors. A phase is performed to limit the value to the quantized range of 0 to 2. bit -1; is represented as: Clip: W_Q = clamp(W_int, 0, N), where N = 2 bit -1;

[0060] (7) is equivalent to subtracting zero from the previously quantized value, which is expressed as: Sub: W_Q = W_Q - Z_W;

[0061] (8) Multiply by the previous scale value. These two steps are equivalent to dequantization, which can be expressed as: Mul: W_float=W_Q*S;

[0062] Therefore, steps (1)-(6) above are performing quantization operations, corresponding to X_Q=clamp(0,N-1,round(X*S)+Z); (7)-(8) are performing dequantization operations, corresponding to X_float=(X_Q-Z) / S; therefore, the quantized S value can be taken out from the Div in step (1), and the Z value is taken out from the second Sub, i.e., step (7). Since this is the quantization of Weight, S_W and Z_W are used to represent it; in the node, division is performed before multiplication, while in the principle, multiplication is performed before division. The inconsistency is because dividing a number is the same as multiplying the reciprocal of a number, so the two are equivalent. Here, S is just a substitute and not a specific value. It is only necessary to ensure that the two are consistent when quantizing and dequantizing. The same principle applies to adding or subtracting zero values.

[0063] like Figure 1 As shown, the first clip node after the conv node is the network's built-in activation (ReLU6) operation after convolution, which is a non-quantization process. The subsequent black nodes represent the quantization steps of the feature, similar to the weight part above, so the corresponding S_F and Z_F can be extracted.

[0064] S3, from the analysis of the quantization method in S1 and the nodes in S2, we can see that the value of S can be obtained from the division Div. The input[0] of the Div before node Conv corresponds to the weight of Conv, the input[1] corresponds to the S_W of Conv, and the input[1] of the Div after Conv corresponds to the scaling factor S_F of Conv.

[0065] The value of Z is obtained in the subtraction Sub. There are two Subs before and after Conv, but only the second Sub of each is the true zero point. Z can be obtained by judging the length of the node's attribute. When the dimension of the node's attribute value node.attribute.t.dims has a value, it is the zero point Z_W; otherwise, it is Z_F.

[0066] S4, finally yielding the simplified ONNX, shows that in addition to the original Conv attributes, the required quantization parameters S_W, S_F, Z_W, and Z_F have been added to the Attributes. When converting to *.mnn format, these parameters will be modified accordingly and read. During the forward pass, the parameters will be extracted for quantization to replace the original quantization method. That is, because quantization in MNN is completed during the conversion, but the current ONNX has already been quantized, i.e., S and Z have been obtained, so there is no need to calculate S and Z again. Only the input data needs to be quantized.

[0067] Furthermore, the method of this application can be described as follows:

[0068] First, let's take a simple linear quantization method as an example:

[0069] Quantize X_int = round(X*S) + Z

[0070] X_Q = clamp(0, N-1, X_int)

[0071] De-Quantize X_float=(X_Q-Z) / S

[0072] Where X is the original value of the float32, S is the scaling factor of the float32, Z is the offset of the float32, and N = 2. n -1, n represents the bit width of quantization, X_Q is the quantized value, and X_float is the dequantized value. Here, it is assumed that quantization has been completed, that is, appropriate S and Z have been obtained, which is also a key step in post-quantization.

[0073] like Figure 1 As shown, this is a linearly quantized convolutional layer (BN has been merged before quantization). The series of nodes above node Conv quantize the weights, and the following nodes quantize the features. The goal is to obtain the quantization parameters, namely the scaling factor S_W and zero point Z_W of the weights, and the scaling factor S_F and zero point Z_F of the features.

[0074] Analytical quantification methods and Figure 1From the nodes in the table, we can see that the value of S can be obtained from the Div. The input[0] of the Div before Conv corresponds to the Weight of Conv, and the input[1] corresponds to the S_W of Conv (divided into channels, values ​​of channels). The input[1] of the Div after Conv corresponds to the S_F of Conv. The value of Z can be obtained from the Sub. Figure 1 As can be seen, there are two Subs at the top and bottom, but only the second Sub is the true zero point (the first Sub calculates the difference between integer and floating-point values). Z can be obtained by judging the length of the node's attribute, and it is Z_W when node.attribute.t.dims has a value (because weight is channel-specific and the value is multi-dimensional), otherwise it is Z_F (feature is channel-independent and has a single value). In addition, in order to ensure the continuity between the nodes of the entire network after removing useless nodes, the output node of Conv is changed to the output node of the last Mul of the quantized feature, so that the Conv of this layer is naturally connected to the next layer.

[0075] The final simplified ONNX diagram is obtained, as follows: Figure 2 As shown, in Attributes, in addition to the original attributes of Conv, the required quantization parameters S_W, S_F, Z_W, and Z_F have been added. When we convert to *.mnn later, we will make simple modifications to extract these parameters, and then extract the parameters again during the forward operation to perform quantization operations to replace the original quantization method.

[0076] For other quantization methods, especially those that require small batches of calibration data to calculate and backpropagate to obtain S, Z, or other quantization parameters, a similar approach can be used. When modifying the method, pay attention to the corresponding parameter extraction and the connection relationship between node inputs and outputs.

[0077] In summary, this application proposes a method for rapidly integrating post-quantization methods into MNNs, which is particularly useful for quantization algorithms with complex quantization processes. The process of obtaining quantization parameters S and Z still follows the original procedure. Afterwards, ONNX is simplified offline, unnecessary nodes are removed, the corresponding parameters are extracted, and then integrated into the MNN. Modification is simple and convenient.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for rapidly combining various quantization algorithms with MNNs, characterized in that, The method includes the following steps: S1, Analysis of linear quantization methods: set up: Quantification ; X_Q = clamp(0, N - 1, X_int) De-quantize X_float = (X_Q - Z) / S Where X is the original value of the float32, S is the scaling factor of the float32, Z is the offset of the float32, and N = 2. n -1, where n represents the quantization bit width, and N is calculated from bits; X_int is the quantized value compared to X_float, X_Q is the quantized value, and X_float is the dequantized value. Here, it is assumed that quantization has been completed, that is, S and Z have been obtained. S2, assuming a linearly quantized convolutional layer, the series of nodes above the Conv node quantizes the weights W, followed by the quantization of the features. The goal is to obtain only the quantization parameters, namely the scaling factor S_W and zeros Z_W of W, and the scaling factor S_F and zeros Z_F of the features; further including the following steps: (1) The original weight is first divided by the S value, which is expressed as: Div: W' = W / S; (2), round() is the rounding function, expressed as: Atten: W_r = round(W'); (3) This step is equivalent to calculating the difference between the integer and the floating point before rounding, which is expressed as: Sub: d = W_r - W'; (4), The result of this step is actually the same as the result after rounding in (2), which is represented as Add: W_int = W' +d; (5) Add the zero point value, which is represented as: Add: W_int = W_int + Z_W; (6) This operation is to prevent out-of-bounds errors. It involves a stage that limits the value to the quantized range of 0 ~ This is represented as: Clip: W_Q = clamp(W_int, 0, N), where N = ; (7) is equivalent to subtracting zero from the previously quantized value, which is expressed as: Sub: W_Q = W_Q - Z_W; (8), Multiply by the previous S value, and it can be expressed as: ; Therefore, steps (1)-(6) above are performing quantization operations, corresponding to X_Q = clamp(0, N - 1, (7)-(8) are performing inverse quantization operations, corresponding to X_float = (X_Q - Z) / S; therefore, the quantized S value can be taken out from Div in step (1), and the Z value is taken out from the second Sub, i.e. step (7). Since this is the quantization of weights, S_W and Z_W are used to represent it. S3, analyzing the quantization method in S1 and the nodes in S2, shows that the value of S can be obtained from the division Div. The input[0] of the Div before node Conv corresponds to the weight, and input[1] corresponds to the S_W of Conv. The input[1] of the Div after Conv corresponds to the scaling factor S_F of Conv. Among them, the division Div has two inputs, input[0] represents the first input, and input[1] represents the second input. The value of Z is obtained in the subtraction Sub. There are two Subs before and after Conv, but only the second Sub of each is the true zero point. Z can be obtained by judging the length of the node's attribute. When the dimension of the node's attribute value node.attribute.t.dims has a value, it is the zero point Z_W; otherwise, it is Z_F. S4, ultimately yielding the simplified ONNX, shows that in addition to the original Conv attributes, the required quantization parameters S_W, S_F, Z_W, and Z_F have been added. This will be used for subsequent conversion. When formatting, make corresponding modifications to read these parameters. When forwarding, read and extract the parameters to perform quantization operations to replace the existing quantization method. That is, because the quantization in MNN is completed during the transformation, but the current ONNX has already been quantized, that is, S and Z have been obtained, so there is no need to calculate S and Z again. Only the input data needs to be quantized.

2. The method for rapidly combining various quantization algorithms and MNNs according to claim 1, characterized in that, In step S2, the convolutional layer of the linear quantization has already had BN merged before quantization.

3. The method for rapidly combining various quantization algorithms and MNNs according to claim 1, characterized in that, In step S3, the input[0] of the Div before the convolution Conv corresponds to the weight of Conv, and the input[1] corresponds to the S_W of Conv, which is divided into channels, with 'channels' values.

4. The method for rapidly combining various quantization algorithms and MNNs according to claim 1, characterized in that, In step S3, there are two Subs before and after Conv, and the first Sub of each is the calculation of the difference between integer and floating-point values.

5. A method for rapidly combining various quantization algorithms and MNNs according to claim 1, characterized in that, In step S3, when the dimension node.attribute.t.dims of the node's attribute value has a value, it is zero point Z_W because the weight is divided into channels and there are multiple values; otherwise, it is Z_F, that is, the Feature does not divide into channels and has a single value.

6. A method for rapidly combining various quantization algorithms and MNNs according to claim 1, characterized in that, In step S3, in order to ensure the continuity between all network nodes after removing useless nodes, the output node of Conv is changed to the output node of the last multiplication Mul of the quantized feature, so that the Conv layer is naturally connected to the next layer.

Images