A method for keeping quantization inference consistent with training end data

Abstract

The application provides a method for keeping consistency of quantization reasoning and training end data, which comprises the following steps: S1, obtaining the weight of float data; S2, obtaining the maximum value of the whole weight data, since the weight data has both positive and negative numbers, the maximum value of the absolute value abs() function of the weight is obtained, the weight is divided by the maximum value, and the numerical distribution is between-1 and 1; S3, when quantizing the data at the GPU training end, the data is subjected to a 4-bit operation; S4, inputting the round() function operation multiplied by 128 values, since the int8 value range is-128 to 127, 128 values are selected for multiplication; S5, performing the last step of the clip(-128, 127) function, finally making the model quantized from float data to discrete data between-128 and 127. Based on the analysis of the inconsistent results of weight quantization, the weight data is processed, the difference between the model reasoning and the training data is reduced, and the correctness of the board end data result is ensured.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a method for maintaining consistency between quantized inference and training data. Background Technology

[0002] As model predictions become more accurate and networks become deeper, the memory consumption of neural networks becomes a problem, especially in mobile applications such as the T31 series chips from Beijing Junzheng Integrated Circuit Co., Ltd. Typically, the capacity of on-board flash memory is very small, and model size is not only a matter of memory capacity but also memory bandwidth. The model uses weights in each prediction, and image-related applications often require real-time data processing. Therefore, low-bit quantization of weights is crucial. Quantization can significantly reduce the model size; quantizing float to 8 bits can reduce the model size by a factor of four, thereby greatly improving the network's operating speed.

[0003] However, during quantization, the network training and inference are performed on different devices (GPU, CPU), resulting in inconsistent quantization results and inconsistencies in the results of the trained model on the board-side inference side.

[0004] In existing technologies, when quantizing weights (float 32-bit), the inconsistency in precision between the GPU and the inference CPU (generally, the inconsistency between GPU and CPU precision exists in the last 7 decimal places) leads to inconsistencies in the quantization results of the model, which increases the error at the inference end.

[0005] In addition, commonly used terms in the prior art include:

[0006] Low-bit quantization: quantizes the weights and features from a 32-bit width (float) to (8-bit, 4-bit, 2-bit).

[0007] Network training: Define the structure of the neural network and the output of forward propagation, define the loss function and select the backpropagation optimization algorithm. Use the BP algorithm to propagate gradients and optimize the network. Run the backpropagation optimization algorithm repeatedly on the training data to make the network adapt to the dataset.

[0008] Quantization inference: The network weights at the training end are fixed, and there is no backpropagation process, so the model can be fixed. At the same time, since the model is fixed during inference, it can be optimized. Normally, floating-point numbers are used for inference, but floating-point inference has high requirements for model size and time. Therefore, the model is quantized into 8-bit integers to accelerate network inference and reduce the model memory usage at the board end.

[0009] Training and inference quantization data consistency: The training end is based on TensorFlow, PyTorch or MXNet related training frameworks to optimize the network on the GPU, and the inference is based on C++ to perform inference on the on-board CPU. Due to the inconsistency between the precision of the GPU and the CPU, and the addition of the round operation during quantization, there is a data inconsistency problem between the training and inference ends, which affects the data inconsistency of the network inference end and reduces the accuracy of the network model during inference.

[0010] The clip() function is used to control the elements in an array within a given range. Given the upper and lower boundaries of the range to be controlled, the clip function changes all values ​​less than the lower boundary to the lower boundary and all values ​​greater than the upper boundary to the upper boundary.

[0011] The round function returns a number that has been rounded to the specified number of decimal places. Summary of the Invention

[0012] To address the aforementioned issues, this method aims to: based on the analysis of inconsistencies in the results during weight quantization, propose processing methods for weight data to reduce the discrepancies between model inference and training data, thereby ensuring the accuracy of the results from the board-end data.

[0013] Specifically, the present invention provides a method for maintaining consistency between quantitative inference and training data, the method comprising the following steps:

[0014] S1, calculate the weights of the float data;

[0015] S2, find the maximum value of the entire weight data. Since the weight data includes both positive and negative numbers, find the maximum value of the absolute value abs() function for the weight. Divide the weight by the maximum value. The values ​​are distributed between -1 and 1.

[0016] S3, when quantizing data on the GPU training side, performs a 4-bit truncation operation on the data;

[0017] S4, input the round() function to multiply by the value of 128. Since the value range of int8 is from -128 to 127, the value of 128 is chosen for multiplication.

[0018] S5 performs the final step of the clip(-128,127) function, which ultimately quantizes the model from float data to discrete data between -128 and 127.

[0019] In step S3, the 4-digit truncation is performed before multiplying by 128 in step S4. The data between -1 and 1 is truncated to 4 decimal places. The 4-digit truncation is chosen because the effective precision of float is 7 digits, so the middle value of 4 is selected. The selection of 4 digits is based on empirical data during model training. This ensures that while maintaining accuracy, it also ensures that there is no inconsistency between the GPU and the inference results during model quantization.

[0020] The method described belongs to the category of weight quantization during low-bit training of deep neural networks.

[0021] The method involves training and inference of the model on different devices during quantization. Since training and inference are not on the same device, the training end runs on a GPU device to accelerate the training of the model; while the inference end runs on a chip-related board device based on the solidified application of the training end model.

[0022] Therefore, the advantage of this application is:

[0023] This method processes the weight data to ensure consistency in data quantization between the network training and inference ends, thereby guaranteeing the correctness of the inference process and results. To reduce the storage and size of the weights, low-bit quantization is performed on the weight data, reducing the network model size and improving network running speed. Attached Figure Description

[0024] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0025] Figure 1 This is a schematic diagram of the weight quantization inference process in existing technology.

[0026] Figure 2 This is a schematic diagram of an example of a weight in this invention.

[0027] Figure 3 This is a schematic diagram of the reasoning process for weight quantization in the method of this invention.

[0028] Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0029] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0030] This application pertains to the quantization of weights during the training process of deep neural networks based on low-bit (4-bit, 5-bit, 8-bit) data. The float data of the weights is quantized to int8, i.e., discrete numbers between (-128, 127). However, since training and inference are not on the same device, the training end is used on a large number of GPU devices (such as NVIDIA GPU devices) to accelerate the training model and significantly reduce the training time. Inference, on the other hand, is based on the solidification of the training model, such as the application on board devices related to the Beijing Junzheng T31 chip. This results in inconsistencies between training and inference data, which greatly leads to errors in the inference model results and significantly increases the error rate of the model inference results.

[0031] This application proposes an additional processing of the weight data during the quantization process based on low-bit quantization of the weights, to ensure the consistency of data between the GPU and the CPU.

[0032] 1. Weight quantification reasoning process: such as Figure 1 As shown:

[0033] First, we find the maximum value of the entire weighted data. Since the weighted data contains both positive and negative numbers, it is roughly distributed as follows: Figure 2 As shown (an example of weights):

[0034] Therefore, the maximum value of abs() is obtained by dividing the weight by the maximum value, and the value is distributed between -1 and 1;

[0035] Then, the data distributed between -1 and 1 are multiplied by a value of 128 (the value range of int8 is -128 to 127, so a value of 128 is chosen for multiplication), which ultimately makes the float data fixed-point between -128 and 128. Since int8 can only be between -128 and 127, the final step is to clip(-128, 127), which finally quantizes the model from float data to discrete data between -128 and 127, greatly reducing the model capacity while maintaining the model accuracy.

[0036] 2. As mentioned earlier, when quantizing the model, since training and inference are performed on different devices, the accuracy of the data on different devices varies. This difference is not obvious for float data, but it will lead to a difference of 1 in the quantization result when quantizing the weights. However, the inference of a model is composed of many layers, which will cause the error to accumulate and become larger and larger.

[0037] Specific examples are as follows:

[0038] GPU training: X * 128 = 0.499999999, the result of the round operation is 0;

[0039] CPU inference: X * 128 = 0.500000001, the result of the round operation is 1;

[0040] The inconsistency between these two results will lead to the unavailability of the results of the inference model and an increase in the error of the inference model results;

[0041] Solution:

[0042] To ensure consistency between the quantized data at the training end and the data at the inference end, the maximum precision of the float model is 7 decimal places. Therefore, when quantizing the data at the training end, the data is truncated by 4 digits before multiplying by 128 using the round() function to ensure consistency between GPU and CPU data. The round() function returns a value that is the result of rounding to the specified number of decimal places. For example, 0.153 * 128 = 19.584, round(19.584) = 20.

[0043] like Figure 3 As shown, before multiplying by 128, the data between -1 and 1 is truncated to 4 decimal places to ensure the consistency of precision between GPU and CPU data, and ultimately ensure the consistency of network inference model results.

[0044] Note: Choosing to retain 4 decimal places ensures accuracy while preventing inconsistencies between GPU and inference results during model quantization. The effective precision of float is 7 digits, so the middle value of 4 digits is chosen.

[0045] This method uses a 4-bit clip to adjust the precision of the weighted data, ensuring the correctness of the inference results.

[0046] In summary, such as Figure 4 As shown, the method includes the following steps:

[0047] S1, calculate the weights of the float data;

[0048] S2, find the maximum value of the entire weight data. Since the weight data includes both positive and negative numbers, find the maximum value of the absolute value of the weight, divide the weight by the maximum value, and the value is distributed between -1 and 1.

[0049] S3, when quantizing data on the GPU training side, performs a 4-bit truncation operation on the data;

[0050] S4, input the round() function to multiply by the value of 128. Since the value range of int8 is from -128 to 127, the value of 128 is chosen for multiplication.

[0051] S5 performs the final step of the clip(-128,127) function, which ultimately quantizes the model from float data to discrete data between -128 and 127.

[0052] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for maintaining consistency between quantitative inference and training data, characterized in that, The method addresses the issue that training and inference of the model occur on different devices during quantization. Since training and inference are not on the same device, the training side runs on a GPU to accelerate model training; while inference is based on the solidified application of the trained model on a chip-related board-side device. The method includes the following steps: S1, calculate the weights of the float data; S2, find the maximum value of the entire weight data. Since the weight data includes both positive and negative numbers, find the maximum value of the absolute value of the weight, divide the weight by the maximum value, and the value is distributed between -1 and 1. S3, when quantizing the data on the GPU training end, the data is truncated to 4 bits; the truncation to 4 bits is to truncate the data between -1 and 1 to 4 decimal places before multiplying by 128 in step S4. The reason for truncating to 4 bits is that the effective precision of float is 7 bits, so the middle value of 4 is chosen. The choice of 4 bits is based on empirical data during model training, which ensures that while ensuring accuracy, it also ensures that there is no inconsistency between the GPU and the inference end results during model quantization. S4, input the round() function to multiply by the value of 128. Since the value range of int8 is from -128 to 127, the value of 128 is chosen for multiplication. S5 performs the final step of the clip(-128,127) function, which ultimately quantizes the model from float data to discrete data between -128 and 127.

2. The method for maintaining consistency between quantized inference and training data according to claim 1, characterized in that, The method described belongs to the category of weight quantization during low-bit training of deep neural networks.

3. The method for maintaining consistency between quantitative inference and training data according to claim 1, characterized in that, In step S1, the absolute value is obtained using the abs() function.

Images