Quantization neural network training method based on gradient scaling correction

Abstract

The invention discloses a quantitative neural network training method based on gradient scaling correction, a neural network model is used for being deployed on an edge device to execute a reasoning task, and the method comprises the following steps: carrying out quantization parameter processing on a pre-trained neural network model based on a post-training quantization method to obtain a quantization model; performing forward propagation on the quantitative model based on a to-be-trained data set; carrying out inverse quantization on an output result of the quantization model, and then carrying out processing based on a loss function and carrying out backward propagation; carrying out scaling processing on each layer of gradient of the quantitative model based on a gradient scaling correction method; and updating network parameters of the quantitative model based on the scaled gradient. According to the training method provided by the invention, gradient distortion and activation distribution abnormity occurring in the neural network training process can be avoided, and effective training of the neural network model can be directly completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network training, and in particular relates to a quantized neural network training method based on gradient scaling correction. Background Art

[0002] Edge computing has become an alternative to cloud computing due to issues such as data privacy, response latency, and energy efficiency in data transmission. Deep neural networks have achieved remarkable results in various fields, which has promoted the development of edge neural network technology. Deep neural networks are usually deployed in drones, surveillance equipment, and mobile phones by loading models in the cloud or downloading trained models from local servers to edge devices. Due to the limited computing resources of edge devices, the models provided by the cloud are usually quantized versions. Even if the trained model is downloaded from the local server, it must be quantized before deployment, and the inference process of the model is usually only performed in the edge device. Therefore, the edge device does not actively update the model parameters to adapt to the external changing environment. When the external environment changes and the model parameters need to be updated, the model parameters need to be adjusted on the cloud server or local server and then downloaded to the edge device. However, sometimes due to data privacy, communication conditions, or to reduce server-side computing requirements, edge devices are required to use data on site instead of sending data to remote locations. Therefore, real-time training on edge devices is more conducive to solving such problems.

[0003] At present, the main methods of quantization training are post-training quantization (PTQ) and quantization-aware training (QAT). The former uses a small data set to quantize the trained model to obtain the quantization parameters. Its quantization speed is fast but it cannot eliminate the quantization error. The latter introduces quantization error during the training process and retrains with a complete data set to eliminate the quantization error and improve accuracy, but it requires a lot of computing resources. However, neither of the two existing methods can directly and effectively train the quantized model. It is usually necessary to dequantize the quantized model into a high-precision model, retrain it, and then quantize it again for deployment. Therefore, there has been no relevant research on directly training the quantized neural network model.

[0004] In view of the above problems, it is urgent to propose a quantized neural network training method based on gradient scaling correction. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a quantized neural network training method based on gradient scaling correction to solve the problems existing in the above prior art.

[0006] To achieve the above object, the present invention provides a quantized neural network training method based on gradient scaling correction, wherein the neural network model is deployed on an edge device to perform reasoning tasks, and the method comprises the following steps:

[0007] The pre-trained neural network model is quantized based on the post-training quantization method to obtain a quantized model;

[0008] Acquire a data set to be trained, and perform forward propagation on the quantization model based on the data set to be trained;

[0009] Dequantizing the output of the quantization model, and then processing and backpropagating based on the loss function;

[0010] Scaling the gradients of each layer of the quantization model based on a gradient scaling correction method;

[0011] The network parameters of the quantization model are updated based on the scaled gradient to complete the training.

[0012] Optionally, the process of performing quantization parameter processing on the pre-trained neural network model based on the post-training quantization method to obtain the quantized model includes:

[0013] The BN layer, convolution layer and activation layer of the pre-trained neural network model are fused, and then the fused neural network model is forward propagated to count the maximum and minimum activation values ​​of each layer and the maximum and minimum weights, and the corresponding quantization parameters are obtained based on the non-uniform unified quantization method to obtain the quantization model.

[0014] Optionally, the formula of the non-uniform unified quantization method is as follows:

[0015]

[0016]

[0017] Where S and Z represent the scaling factor and zero point respectively; q and x represent the quantized value and the original value respectively; x min and x max The minimum and maximum values ​​of the original value respectively; bit indicates the number of quantization bits; round indicates rounding operation; clamp indicates truncating x to [x min ,x max ] range.

[0018] Optionally, the process of forward propagating the quantization model based on the data set to be trained is completed based on an integer type.

[0019] Optionally, the loss function is a combination of a cross entropy loss function and an activation loss function.

[0020] Optionally, the formula of the activation loss function is as follows:

[0021] Loss = Lce +L act

[0022]

[0023] In the formula, β i is the variable parameter of each network layer; n is the last n neural network layers; A and Z represent the output activation and corresponding quantization parameters of the corresponding layer, respectively, p ij is the activation value of the i-th layer, m i is the total number of activation values ​​of the i-th layer, and bit is the quantized bit width.

[0024] Optionally, the formula of the gradient scaling correction method is as follows:

[0025]

[0026] In the formula, g c and g fc are the gradients of the convolutional layer and the fully connected layer, respectively; |·| max Indicates the maximum absolute value; BL i and BL represent the i-th residual block and the total number of residual blocks respectively; e and epoch are the current iteration number and the total number of iterations respectively; α is a constant coefficient; bit represents the number of quantization bits.

[0027] The present invention also provides an electronic device, comprising: a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement each step of the quantized neural network training method based on gradient scaling correction.

[0028] The present invention also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the various steps of the quantized neural network training method based on gradient scaling correction are implemented.

[0029] Compared with the prior art, the present invention has the following advantages and technical effects:

[0030] The present invention processes the quantization parameters of the pre-trained neural network model based on the post-training quantization method to obtain the quantization model; forward propagates the quantization model based on the data set to be trained; dequantizes the output result of the quantization model, and then processes and propagates it backward based on the loss function; scales the gradients of each layer of the quantization model based on the gradient scaling correction method; and updates the network parameters of the quantization model based on the scaled gradients. The training method proposed by the present invention can avoid the gradient distortion and activation distribution abnormality that occur in the neural network training process, and can directly complete the effective training of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 This is a flow chart of a quantized neural network training method based on gradient scaling correction according to an embodiment of the present invention;

[0033] Figure 2 It is a quantized neural network activation distribution anomaly graph according to an embodiment of the present invention;

[0034] Figure 3 A ratio curve diagram of a quantized neural network weight norm and a gradient norm according to an embodiment of the present invention;

[0035] Figure 4 This is an activation distribution diagram after adding activation loss in an embodiment of the present invention;

[0036] Figure 5 This is a curve diagram of the ratio of the weight norm to the gradient norm after gradient scaling correction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Embodiment 1

[0040] In the public data sets (Cifar10, Cifar100, Flower102, Mini-Imagenet, etc.), the accuracy of the quantized ResNet18 model was significantly reduced by fine-tuning. Analysis found that the reasons for the decline were: gradient distortion and abnormal activation distribution. In this regard, the present invention provides a quantized neural network training method based on gradient scaling correction to achieve the purpose of effectively training the quantized neural network directly. Among them, the neural network model is used to be deployed on edge devices to perform inference tasks, such as Figure 1 As shown, the method comprises the following steps:

[0041] The pre-trained neural network model is quantized based on the post-training quantization method to obtain a quantized model;

[0042] Acquire a data set to be trained, and perform forward propagation on the quantization model based on the data set to be trained;

[0043] Dequantizing the output of the quantization model, and then processing and backpropagating based on the loss function;

[0044] Scaling the gradients of each layer of the quantization model based on a gradient scaling correction method;

[0045] The network parameters of the quantization model are updated based on the scaled gradient to complete the training.

[0046] It is feasible to perform quantization parameter processing on the pre-trained neural network model based on the post-training quantization method, and the process of obtaining the quantized model includes:

[0047] The BN layer of the model is fused with the convolution layer and the activation layer, and then a small amount of data is used to input the pre-trained model for forward calculation. During the calculation process, the minimum and maximum activation values ​​of each layer and the minimum and maximum weights are counted, and the corresponding quantization parameters are calculated using the non-uniform unified quantization method to obtain the quantization model.

[0048] As an additional embodiment, non-uniform uniform quantization is shown in formula (1-3).

[0049]

[0050] Where S and Z represent the scaling factor and zero point respectively; q and x represent the quantized value and the original value respectively; x min and x max The minimum and maximum values ​​of the original value respectively; bit indicates the number of quantization bits; round indicates rounding operation; clamp indicates truncating x to [x min ,x max ] range.

[0051] It is feasible to input data into the quantized network for forward propagation, and the calculation process is implemented only using integers (int8), as shown in the following formula:

[0052]

[0053] The feasible process of calculating the loss function includes:

[0054] Since the quantization model usually integrates the BN layer, it is easy for the model to have abnormal activation distribution during training, resulting in reduced accuracy, such as Figure 2As shown in the figure, the activation distribution changes rapidly at the beginning of training, and significant truncation occurs at the 10th iteration, and the normal distribution trend gradually disappears. The BN layer helps to adjust the activation distribution during the forward propagation process and enhance the generalization ability of the model. The quantization model incorporates the BN layer, which is equivalent to a network model without a BN layer. If the abnormal activation distribution during the training process can be suppressed, the generalization ability of the model can be improved to a certain extent. The loss function commonly used in classification prediction models is the cross entropy loss function (Cross Entropy Loss), which only acts on the classification results and has no effect on suppressing abnormal activation distribution. Therefore, this embodiment proposes to add an activation loss function (AL) term to suppress the movement of activation to the cutoff area during training, thereby further improving the prediction accuracy of the model.

[0055] Loss = L ce +L act (5)

[0056]

[0057] In the formula, β i is a variable parameter of each network layer. The more abnormal the activation distribution of the layer is, the larger the value is. n represents the last n neural network layers. A and Z represent the output activation of the layer and the corresponding quantization parameter (zero point) respectively. p i j is the activation value of the i-th layer, m i is the total number of activation values ​​of the i-th layer, and bit is the number of quantization bits.

[0058] It is feasible to use the gradient scaling correction (GSC) method to scale the gradient to a specified size.

[0059] The training of the quantized model is different from that of the full precision model. Its parameters are all integers (int8), and due to the rounding operation in the forward propagation process, the mismatched gradients are difficult to effectively update the quantized model parameters. Figure 3 As shown in the figure, a ratio curve of the weight norm and the gradient norm is plotted based on the ResNet18 model. The uncorrected curve shows that its gradient is more than 5 orders of magnitude smaller than the weight. After correction using the existing QAS method, the gradients of some network layers still do not match the weights, which makes it difficult to directly train the quantized model. Considering the particularity of the quantized model parameters, it is expected that the corrected ratio falls within the expected range. In this regard, the present invention proposes a new gradient scaling correction method.

[0060]

[0061] In the formula, g c and g fc are the gradients of the convolutional layer and the fully connected layer, respectively; |·|max Indicates the maximum absolute value; BL i and BL represent the i-th residual block and the total number of residual blocks respectively; e and epoch are the current iteration number and the total number of iterations respectively; α is a constant coefficient.

[0062] Finally, the quantization model parameters are updated according to the corrected gradients.

[0063] Embodiment 2

[0064] Taking the mini-imagenet dataset as an example, ResNet18 is used as the basic model for prediction and classification tasks. The network model has 18 layers. This dataset is an imagenet public dataset with 60,000 images divided into 100 categories. The training process uses the SGD optimizer, the learning rate is set to 0.0001, the input image size is adjusted to 128*128 pixels, the weight and activation quantization is 8 bits, and the number of training iterations is 50.

[0065] The data is preprocessed: 80% is used as a training set and 20% is used as a test set. The ResNet18 pre-trained model is loaded. The BN layer and the convolution layer are fused according to formula (10).

[0066]

[0067] In the formula, γ and β are the trainable parameters of the BN layer; μ and δ are the mean and variance of the statistical input data respectively; ε is a very small positive number to avoid the denominator being 0; w and b are the weights and biases of the original convolutional layer. After integrating the BN layer, the new weight is Bias is

[0068] Input the data into the fused model, count the maximum and minimum values ​​of the input and output activations of each layer, and calculate the activation quantization parameters. Calculate the weight quantization parameters according to formula (1-3), and quantize the weights to 8-bit integers to obtain a quantized model.

[0069] The data is input into the quantization model and forward propagated according to formula (4). In order to avoid overflow problems, the activation value of each network layer needs to be truncated to [-127, +127] during the calculation process.

[0070] In order to avoid data overflow when calculating the loss function, the final prediction output of the model is first dequantized according to formula (11), and then the loss function is calculated. The loss function consists of two parts, one is the cross entropy loss, such as formula (12); the other is the activation loss, such as formula (6-7). At this time, only the activation distribution of the last convolutional layer is suppressed, and the β of this layer is calculated according to formula (7): 18, and calculate the activation value of the layer minus the quantization parameter zero point Z of the layer according to formula (6), and then calculate the average of the absolute values, so as to calculate the activation loss of the last layer. Finally, according to formula (5), the two parts of the loss function are added to obtain the final loss function, and the gradient is back-propagated.

[0071]

[0072] Where y is the true label, Predict output for the model.

[0073] like Figure 4 As shown in the figure, taking the ResNet18 model as an example, after adding activation loss and training for 50 cycles, the activation value shows a normal distribution as a whole compared with that without adding activation loss.

[0074] Traverse the network layers from back to front and correct the gradient according to formula (8-9). The fully connected layer is corrected according to formula (9), and the remaining convolutional layers are corrected according to formula (8).

[0075] like Figure 5 As shown, taking the ResNet18 model as an example, the gradient can be scaled to a specified size to match the weight.

[0076] According to the formula of the SGD optimizer, the parameters are updated using the corrected gradients.

[0077]

[0078] Where W t+1 is the updated gradient; W t is the gradient before update; η is the learning rate; is the corrected gradient value. Repeat the above steps until it is iterated 50 times. After the training is completed, the trained model parameters are saved and used for testing on the test set. The test accuracy is 81.56%. Under the same training set and test set, the accuracy of the full-precision model is 81.87%. That is, the present invention can be directly trained on the quantized model and has an accuracy comparable to that of the full-precision model.

[0079] Embodiment 3

[0080] This embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement each step of the quantized neural network training method based on gradient scaling correction.

[0081] Embodiment 4

[0082] This embodiment also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the quantized neural network training method based on gradient scaling correction is implemented.

[0083] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A quantized neural network training method based on gradient scaling correction, wherein: The neural network model is used to be deployed on an edge device to perform reasoning tasks, and the method includes the following steps: The pre-trained neural network model is quantized based on the post-training quantization method to obtain a quantized model; Acquire a data set to be trained, and perform forward propagation on the quantization model based on the data set to be trained; Dequantizing the output of the quantization model, and then processing and backpropagating based on the loss function; Scaling the gradients of each layer of the quantization model based on a gradient scaling correction method; The network parameters of the quantization model are updated based on the scaled gradient to complete the training.

2. The method according to claim 1, characterized in that The process of quantizing the parameters of the pre-trained neural network model based on the post-training quantization method to obtain the quantized model includes: The BN layer, convolution layer and activation layer of the pre-trained neural network model are fused, and then the fused neural network model is forward propagated to count the maximum and minimum activation values ​​of each layer and the maximum and minimum weights, and the corresponding quantization parameters are obtained based on the non-uniform unified quantization method to obtain the quantization model.

3. The method according to claim 2, characterized in that The formula of the non-uniform unified quantization method is as follows: Where S and Z represent the scaling factor and zero point respectively; q and x represent the quantized value and the original value respectively; x min and x max The minimum and maximum values ​​of the original value respectively; bit indicates the number of quantization bits; round indicates rounding operation; clamp indicates truncating x to [x min ,x max ] range.

4. The method according to claim 1, characterized in that: The process of forward propagating the quantization model based on the data set to be trained is completed based on the integer type.

5. The method according to claim 1, characterized in that The loss function is a combination of the cross entropy loss function and the activation loss function.

6. The method according to claim 5, characterized in that The formula of the activation loss function is as follows: Loss=L ce +L act In the formula, β i is the variable parameter of each network layer; n is the last n neural network layers; A and Z represent the output activation and corresponding quantization parameters of the corresponding layer, respectively, p ij is the activation value of the i-th layer, m i is the total number of activation values ​​of the i-th layer, and bit is the number of quantization bits.

7. The method according to claim 1, characterized in that The formula of the gradient scaling correction method is as follows: In the formula, g c and g fc are the gradients of the convolutional layer and the fully connected layer, respectively; |·| max Indicates the maximum absolute value; BL i and BL represent the i-th residual block and the total number of residual blocks respectively; e and epoch are the current iteration number and the total number of iterations respectively; α is a constant coefficient; bit represents the number of quantization bits.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the quantized neural network training method based on gradient scaling correction as described in any one of claims 1 to 7.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the quantized neural network training method based on gradient scaling correction as described in any one of claims 1 to 7 are implemented.

Images