Method for rapidly analyzing quantification effect
A method for neural network quantization using cosine similarity and absolute error analysis efficiently identifies and resolves conversion issues, ensuring high accuracy and minimal error in INT8 conversion.
Patent Information
- Application Number
- CN202310737881.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2024-12-20
AI Technical Summary
Existing neural network quantization methods require manual layer fusion and dependency resolution, lacking efficient tools to quickly identify problem points during the conversion from FP32 to INT8, leading to inefficiencies in locating and addressing issues.
A method involving training a floating-point model, calculating cosine similarity and maximum absolute error for each layer, using quantization formulas to convert to INT8, and comparing results to identify and resolve quantization errors.
Enables rapid identification and resolution of quantization issues, ensuring high cosine similarity (>99%) and minimal absolute error for efficient 8-bit conversion, with empirical thresholds guiding problem detection.
Smart Images
Figure CN119166098A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of neural network processing, and in particular relates to a method for quickly analyzing quantization effects. Background Art
[0002] As model predictions become more accurate and networks become deeper, the memory size consumed by neural networks becomes a problem. The industry eventually chose INT8 quantization - FP32 is replaced by INT8 during inference, while training remains FP32. The quantization process can be divided into two parts: converting the model from FP32 to INT8, and using INT8 for inference.
[0003] However, when quantizing the model on the PC side, it is often necessary to manually fuse the layers and then insert quantization nodes, such as conv+bn+act (convolution+BatchNorm+activation), add+act (addition+activation), etc. Because most neural networks are serial, there will be dependencies between the upper and lower layers, and sometimes it is necessary to quickly locate the problem point.
[0004] In addition, commonly used technical terms include:
[0005] 1. Float model: refers to the original model of the neural network. The conventional precision model generally indicates that the model weight value format is FP32 (32-bit floating point, single precision), hereinafter referred to as the float model;
[0006] 3. Model quantization refers to converting the floating-point algorithm of the neural network into fixed-point. A low-precision model means that the model weight value format is FP16 (half-precision floating point) or INT8 (8-bit fixed-point integer), but currently low precision often refers to INT8.
[0007] 4. Cosine similarity, also known as cosine similarity, evaluates the similarity of two vectors by calculating the cosine value of the angle between them. The closer the cosine value is to 1, the closer the angle between the two vectors is to 0 degrees, and the more similar the two vectors are. If there are two vectors in n-dimensional space, vector a (a1, a2, a3, ...., an) and vector b (b1, b2, b3, .... bn), according to the dot product formula, we can get:
[0008]
[0009] 4. Maximum absolute error: find the maximum value of the difference between the absolute values of the two neural network output layer tensors. Summary of the invention
[0010] In order to solve the above problems, the purpose of this application is to propose a practical and effective method to analyze the quantized network layers compared with the original float, so as to quickly view the effect of the quantized model and locate the layer with problems.
[0011] Specifically, the present invention provides a method for quickly analyzing quantization effects, the method comprising: S1. training a floating-point model to obtain a floating-point model with satisfactory accuracy, that is, training a certain number of times until the model converges, that is, the loss remains basically unchanged and the accuracy no longer increases;
[0012] S2. Prepare the calibration set, perform statistics to obtain the minmax required for quantization, and determine the dynamic range of floating-point numbers;
[0013] S3. Convert each value of the weight tensor from FP32 to a low-bit model using the following quantization method. The quantization formula involved is as follows:
[0014]
[0015]
[0016]
[0017] x quantized =x float ÷x scale +x zero_point (4)
[0018] in, Indicates the calculated minimum value of the floating point range, Indicates the maximum value of the calculated floating-point range. Indicates calculating the minimum value of the range you want to quantize. Indicates the maximum value of the range to be quantized, x scale represents the scaling factor, x zero_point The floating point 0 corresponds to the quantized value, and the final formula (4) represents the floating point value x float After quantization, the quantized value x is obtained quantized Calculation method of
[0019] S4. Given the same input, calculate the float and low-bit cosine similarity and maximum absolute error value for the output tensor of each layer of the model:
[0020] cosine_sim=np.dot(ai,bi) / (np.sqrt(np.sum(ai*ai))*np.sqrt(np.sum(bi*bi))), mabs_error=np.max(np.abs(ai-bi))
[0021] Among them, ai and bi are the results of running the float model of the i-th layer and the results of running the low-bit model respectively. Theoretically, the same size means that there is no loss before and after quantization;
[0022] Because sometimes the cosine similarity alone is not absolutely representative, the maximum absolute error is added as a synchronization reference. When the cosine similarity of the two is high and the maximum absolute error is small, it means that the two are very similar, and double protection can be achieved.
[0023] The method can obtain a cosine similarity cosine_sim of each layer of most models quantized to low bit precision lossless values above 99%. Since the cosine similarity cos_sim value ranges from 0 to 1, the closer it is to 1, the smaller the difference between the two. The smaller the maximum absolute error, the closer the pixel values are.
[0024] The lower bits are 8 bits.
[0025] The method directly converts the 8-bit model according to the converged full-precision model, calculates the cosine similarity and absolute error of each layer of 32-bit and 8-bit during the conversion process, and uses the two as references to check the quantized conditions of each layer. The checking process refers to comparing the cosine similarity and maximum absolute error calculated for each layer. If there is an abnormality, the following method is used to locate the layer where the problem begins in time, and then proceed in sequence after the problem is solved. That is, assuming that when a certain layer is observed, cosine_sim suddenly drops very low, that is, less than 90%, and the maximum absolute error also becomes very large, that is, greater than 20. At this time, when the printed value, the cosine similarity is small, or the maximum absolute error is large, it is necessary to check whether the operator corresponding to this layer has a quantization problem that causes the above error. After the problem is solved, step S4 is repeated to calculate and compare the cosine similarity or maximum absolute error of each layer, and a similar result with a high similarity as before will appear.
[0026] The cosine similarity is also called cosine similarity, which evaluates the similarity of two vectors by calculating the cosine value of the angle between them. The closer the cosine value is to 1, the closer the angle between the two vectors is to 0 degrees, and the more similar the two vectors are. If there are two vectors in n-dimensional space, vector a (a1, a2, a3, ...., an) and vector b (b1, b2, b3, .... bn), according to the dot product formula, we can get:
[0027]
[0028] Therefore, the advantages of this application are:
[0029] This method is simple. During the conversion process, the cosine similarity and absolute error of each layer of 32-bit and 8-bit are calculated. The two are used as references to observe the situation of each layer after quantization. If there is an abnormality, this method can also be used to promptly locate the layer where the problem begins, and then proceed in sequence after the problem is resolved. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0031] Figure 1 is a schematic diagram of the process of this method.
[0032] Figure 2 It is a schematic diagram of the mapping relationship of the quantization formula in this method.
[0033] Figure 3 It is a schematic diagram of the effect that the cosine similarity cosine_sim of each layer in the specific embodiment of the method is above 99% and the maximum absolute error is also very small.
[0034] Figure 4 It is a schematic diagram of the effect that in the specific embodiment of the method, it is observed that at the trace_24 layer, cosine_sim suddenly drops very low (78.57%), and the maximum absolute error also becomes very large (319.728). DETAILED DESCRIPTION
[0035] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0036] like Figure 1 As shown, the present application provides a method for quickly analyzing the quantitative effect, comprising the following steps:
[0037] S1. Train the floating-point model to obtain a floating-point model with satisfactory accuracy. The standard of satisfactory accuracy is different for different training sets and network structures of different sizes. It is only necessary to train a certain number of times until the model converges, that is, the loss remains basically unchanged and the accuracy does not increase any more.
[0038] S2. Prepare the calibration set and perform statistics to obtain the minmax required for quantization to determine the dynamic range of floating-point numbers. Currently, the most commonly used methods in major deep learning frameworks include the maximum and minimum values (MinMax), the moving average maximum and minimum values (MovingAverageMinMax) and the KL distance (Kullback-Leibler divergence). The maximum and minimum value method is used here;
[0039] S3. Convert each value of the weight tensor from FP32 to a low-bit model (mainly 8-bit) according to the following quantization method. The quantization formula involved is as follows:
[0040]
[0041]
[0042]
[0043] x quantized =x float ÷x scale +x zero_point (4)
[0044] in, Indicates the calculated minimum value of the floating point range, Indicates the maximum value of the calculated floating-point range. Indicates the minimum value of the range to be quantized (such as 0). Indicates the maximum value of the range you want to quantize (such as 255), x scale represents the scaling factor, x zero_point The floating point 0 corresponds to the quantized value, and the final formula (4) represents the floating point value x float After quantization, the quantized value x is obtained quantized The calculation method, such as Figure 2 As shown;
[0045] S4. Given the same input, calculate the float and 8bit cosine similarity and maximum absolute error of the output tensor of each layer of the model:
[0046] cosine_sim=np.dot(ai,bi) / (np.sqrt(np.sum(ai*ai))*np.sqrt(np.sum(bi*bi)))mabs_error=np.max(np.abs(ai-bi))
[0047] Among them, ai and bi are the results of the operation of the float model of the i-th layer and the 8-bit model respectively. In theory, the same size (shape) means there is no loss before and after quantization. The similarity value is between 0 and 1. The closer to 1, the smaller the difference between the two. Because sometimes the cosine similarity alone cannot be absolutely representative, the maximum absolute error is also added as a synchronization reference. That is, how to judge what conclusion based on this value? When the cosine similarity of the two is high and the maximum absolute error is very small, it means that the two are very similar, double protection.
[0048] For most models quantized to 8-bit lossless precision, the above steps can get the following results, that is, the cosine similarity cosine_sim of each layer is above 99%, and the value of cosine similarity is in the range of 0-1. The larger the value, the closer it is. The maximum absolute error is also small, that is, around 1. The maximum absolute error refers to the maximum value of the absolute value of the error between the corresponding pixels of the current layer. This may be less than 1, or it may be tens to hundreds. The smaller the value, the closer it is. Figure 3 shown.
[0049] If there is an abnormality, the cosine similarity will be above 99%, and the maximum absolute error will be within a few pixel values. Generally, an empirical value of no more than 20 is considered normal.
[0050] However, for some models, the following may occur: Figure 4 The effect shown can be observed in the trace_24 layer. Assume that cosine_sim suddenly drops very low (78.57%) and the maximum absolute error also becomes very large (319.728). At this time, it is necessary to check whether the operator corresponding to this layer has a quantization problem that causes the above error. After solving it, repeat step S4 to calculate and compare the cosine similarity or maximum absolute error of each layer, and a result with a higher similarity similar to the previous one will appear.
[0051] Of course, the above process can only be used as a preliminary reference to quickly verify the 8-bit quantization effect or determine which layer has a problem. Further actual testing of the accuracy of the verification set is required to truly determine the effect of the quantization.
[0052] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for rapid analysis of quantitative effects, characterized in that: The method comprises: S1. Train the floating-point model to obtain a floating-point model with satisfactory accuracy, that is, train a certain number of times until the model converges, that is, the loss remains basically unchanged and the accuracy no longer increases; S2. Prepare the calibration set, perform statistics to obtain the minmax required for quantization, and determine the dynamic range of floating-point numbers; S3. Convert each value of the weight tensor from FP32 to a low-bit model using the following quantization method. The quantization formula involved is as follows: x quantized =x float ÷x scale +x zero_point (4) in, Indicates the calculated minimum value of the floating point range, Indicates the maximum value of the calculated floating-point range. Indicates calculating the minimum value of the range you want to quantize. Indicates the maximum value of the range to be quantized, x scale represents the scaling factor, x zero_point The floating point 0 corresponds to the quantized value, and the final formula (4) represents the floating point value x float After quantization, the quantized value x is obtained quantized Calculation method of S4. Given the same input, calculate the float and low-bit cosine similarity and maximum absolute error value for the output tensor of each layer of the model: cosine_sim=np.dot(ai,bi) / (np.sqrt(np.sum(ai*ai))*np.sqrt(np.sum(bi*bi))), mabs_error=np.max(np.abs(ai-bi)) Among them, ai and bi are the results of running the float model of the i-th layer and the results of running the low-bit model respectively. Theoretically, the same size means that there is no loss before and after quantization; Because sometimes the cosine similarity alone is not absolutely representative, the maximum absolute error is added as a synchronization reference. When the cosine similarity of the two is high and the maximum absolute error is small, it means that the two are very similar, and double protection can be achieved.
2. The method for rapid analysis and quantification effect according to claim 1, characterized in that: The method can obtain a cosine similarity cosine_sim of each layer of most models quantized to low bit precision lossless values above 99%. Since the cosine similarity cos_sim value ranges from 0 to 1, the closer it is to 1, the smaller the difference between the two. The smaller the maximum absolute error, the closer the pixel values are.
3. The method for rapid analysis and quantification effect according to claim 1, characterized in that: The lower bits are 8 bits.
4. The method for rapid analysis and quantification effect according to claim 3, characterized in that: The method directly converts the 8-bit model based on the converged full-precision model. During the conversion process, the cosine similarity and absolute error of each layer of 32-bit and 8-bit are calculated. The two are used as references to check the quantized conditions of each layer. The checking process refers to comparing the cosine similarity and maximum absolute error calculated for each layer. If there is an abnormality, the following method is used to locate the layer where the problem begins in time, and then proceed in sequence after solving it, that is, Assume that when a certain layer is observed, cosine_sim suddenly drops very low, that is, less than 90%, and the maximum absolute error also becomes very large, that is, greater than 20. At this time, when the printed value, the cosine similarity is too small, or the maximum absolute error is too large, it is necessary to check whether the operator corresponding to this layer has a quantization problem that causes the above error. After solving it, repeat step S4, calculate and compare the cosine similarity or maximum absolute error of each layer, and a result with a higher similarity similar to the previous one will appear.
5. The method for rapid analysis and quantification effect according to claim 1, characterized in that: The cosine similarity is also called cosine similarity, which evaluates the similarity of two vectors by calculating the cosine value of the angle between them. The closer the cosine value is to 1, the closer the angle between the two vectors is to 0 degrees, and the more similar the two vectors are. If there are two vectors in n-dimensional space, vector a (a1, a2, a3, ...., an) and vector b (b1, b2, b3, .... bn), according to the dot product formula, we can get: