A Quantization Method for Optimizing Neural Network Model Activations
By fine-tuning the low bit model based on the full-precision model in a convolutional neural network, and stating the maximum feature map value through sliding average, the problem of accuracy loss and difficulty in convergence during training of low bit model is solved, and the convergence speed and effect of the model are improved .
Patent Information
- Application Number
- CN202011617711.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-12-31
AI Technical Summary
The low-bit models in the prior art have serious accuracy losses during training and are difficult to converge, and cannot effectively meet the finite bit width limit of the numerical range during training.
Using a method based on the fine-tuning of the full-precision model, first train the 4-bit model, and then train the 2-bit model. By sliding average statistics on the maximum value of the feature map during each training, we ensure that each layer activation function obtains the most appropriate actual distribution under the current bit width. Maximum value.
It reduces training difficulty, improves the convergence speed and final effect of the model, and ensures the accuracy and performance of the low-bit model.
Smart Images

Figure CN114692814B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of convolutional neural network acceleration, and in particular to a quantization method for optimizing the activation of a neural network model. Background Art
[0002] In recent years, with the rapid development of science and technology, the era of big data has arrived. Deep learning uses deep neural networks (DNNs) as models and has achieved remarkable results in many key areas of artificial intelligence, such as image recognition, reinforcement learning, and semantic analysis. As a typical DNN structure, convolutional neural networks (CNNs) can effectively extract the hidden features of images and accurately classify images. They have been widely used in the fields of image recognition and detection in recent years.
[0003] However, in the prior art, the ReLU function is often used in training full-precision models. Since the real number range represented by full-precision numbers is very wide, it can meet the numerical range required in the training process. However, when training low bits, due to the limitation of bit width, its representation range is limited, resulting in the inability of the model to converge effectively during the training process, and the accuracy of the final model is not ideal.
[0004] Common technical terms in the prior art include:
[0005] Convolutional Neural Networks (CNN): A type of feedforward neural network that includes convolution calculations and has a deep structure.
[0006] Quantization: Quantization refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number (or fewer) of discrete values.
[0007] Low bit: Quantize the data into data with a bit width of 8 bits, 4 bits or 2 bits.
[0008] BN (Batch Normalization): The batch normalization operation can be regarded as a special neural network layer, which is added before each layer of nonlinear activation function.
[0009] Relu (Rectified Linear Unit) operation: also known as the rectified linear unit, is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants. Summary of the invention
[0010] In order to solve the above problems, the present method aims to overcome the defects existing in the above-mentioned prior art and solve the problems of serious accuracy loss and difficulty in convergence of existing low-bit (e.g., 2-bit) models during training.
[0011] This application is to fine-tune a low-bit model based on a full-precision model: first, a full-precision model is trained with a data set to achieve the target accuracy, and then a low-bit model is fine-tuned and trained based on the full-precision model.
[0012] Specifically, the present invention proposes a quantization method for optimizing the activation of a neural network model. The method first trains a 4-bit model based on a full-precision model, and then trains a 2-bit model based on the 4-bit model. When training the 4-bit and 2-bit low-bit models, the maximum values of the feature maps are counted using a sliding average and are updated each time training is performed without relying on the statistical results of the previous model.
[0013] The method comprises the following steps:
[0014] S1: First train a full-precision model based on the dataset to achieve the target accuracy;
[0015] S2, based on the full-precision model training weights and activations quantized to 4 bits, and the maximum value of the feature map is counted during training;
[0016] S3, based on the 4-bit model trained in step S2, the weights and activations are quantized to a 2-bit model, and the maximum value of the feature map is recalculated.
[0017] The step S1 further comprises:
[0018] S1.1, determine the training data:
[0019] The dataset for training the model is ImageNet1000, which is a subset of the ImageNet dataset, with 1.2 million training sets, 50,000 validation sets, 150,000 test sets, and 1,000 categories;
[0020] S1.2, determine the training model:
[0021] The basic neural network model used for training is MobileNetV1, which is a model based on deep separable convolution;
[0022] S1.3, select activation function:
[0023] In the MobileNetV1 model, batch normalization (BN) and activation function ReLU operations are added after each convolution layer.
[0024] S1.4, training network:
[0025] The basic steps for network training are: first use the adam optimizer to train for 60 epochs, and then use the SGD optimizer until the training is completed;
[0026] S1.5, test network effect:
[0027] Use the test set to test the network results.
[0028] In step S1.3, since the trained model needs to be quantized to a low bit and the feature map needs to be quantized to 2 bits afterwards, the activation function Relu operation can also replace the Relu activation function with ReluX during training, as shown in Formula 1:
[0029]
[0030] The step S2 further comprises:
[0031] S2.1, data quantization: quantize the data to be quantized according to formula 2 to obtain low-bit data. During training, quantize the weights and activations to 4 bits:
[0032]
[0033] Variable Description: W f For full precision data, it is an array. q For the simulated quantized data, max w Full precision data W f Maximum value, min w Full precision data W f The minimum value in , b is the bit width after quantization;
[0034] S2.2, while training the model, obtain the maximum value of the feature map and then use the sliding average method to calculate the maximum value, which is represented by maxValue;
[0035] S2.3, the parameter maxValue obtained by the activation function of each layer is updated by the sliding average method, as shown in Formula 3:
[0036] v t =β·v t-1 +(1-β)·(θ t ) Formula 3
[0037] Variable description: v t is the value of variable v at time t, β is the weighting coefficient, θ t is the value of variable v at time t, v t-1is the value of the variable v at time t-1.
[0038] The step S2.2 obtains the maximum value by processing the following steps:
[0039] 1: Assign ReluX(v) to parameter v: v = ReluX(v);
[0040] 2: getChannelsNum(v) gets the number of channels of the feature map and assigns it to the parameter channels: channels = getChannelsNum(v);
[0041] 3: getBatchNum(v) gets the size of the feature map batch and assigns it to the parameter batchNum: batchNum = getBatchNum(v);
[0042] 4: Initialize parameter maxValue: maxValue = 0.0;
[0043] 5: When tag is from 0 to batchNum value, perform: for tag=0to batchNum do;
[0044] 6: vValueTag = v[tag];
[0045] 7:getChannelMax(v) gets the maximum value of each channel in the feature map and assigns it to the parameter perChannelMax:perChannelMax=getChannelMax(vValueTag)
[0046] 8:reduceSum(perChannelMax) calculates the sum of the variable perChannelMax and divides it by the parameter channels value, adds it to the parameter maxValue value and assigns it to the parameter maxValue: maxValue+=reduceSum(perChannelMax) / channels;
[0047] 9:end for;
[0048] 10: Divide the parameter maxValue value by the batchNum value and assign it to the parameter maxValue: maxValue = maxValue / batchNum.
[0049] In step S2.3, the weight coefficient β is set to 0.996 during sliding average, and the initial value is set to 3.0, where the initial value is v in formula 3. tAnd t=0; then during the training process, the value of X is maxValue, and maxValue is recorded. During the inference process, the statistical maximum value is used without recalculating the maximum value of the feature.
[0050] The step S3 comprises:
[0051] S3.1, data quantization: quantize the data to be quantized according to formula 2 to obtain low-bit data. During training, quantize the weights and activations to 2 bits:
[0052]
[0053] Variable Description: W f For full precision data, it is an array. q For the simulated quantized data, max w Full precision data W f Maximum value, min w Full precision data W f The minimum value in , b is the bit width after quantization;
[0054] S3.2, while training the model, obtain the maximum value of the feature map and then use the sliding average method to calculate the maximum value, which is represented by maxValue;
[0055] S3.3, the parameter maxValue obtained by the activation function of each layer is updated by the sliding average method, as shown in Formula 3:
[0056] v t =β·v t-1 +(1-β)·(θ t ) Formula 3
[0057] Variable description: v t is the value of variable v at time t, β is the weighting coefficient, θ t is the value of variable v at time t, v t-1 is the value of the variable v at time t-1.
[0058] When training the 2-bit model, maxValue needs to be recalculated, and the weight coefficient β is set to 0.998 during sliding average, and the initial value is set to 2.0, where the initial value is v in formula 3. t And t=0.
[0059] The method may further comprise:
[0060] S4, test network effect;
[0061] S5, output network.
[0062] Therefore, the advantages of this application are:
[0063] (1) First, train a full-precision model based on the data set, and then train a 4-bit model and a 2-bit model to reduce the training difficulty and improve the model's convergence speed and final effect;
[0064] (2) Since the distribution of the feature map changes each time the bit width is changed, the value of the feature map is recalculated each time the bit width is changed during training, so that each layer can obtain a maximum value that best matches the actual distribution under the current bit width. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.
[0066] Figure 1 It is a schematic diagram of the process of the present invention.
[0067] Figure 2 It is a flowchart of BN and Relu operations.
[0068] Figure 3 It is a flowchart of replacing the Relu activation function with ReluX during training.
[0069] Figure 4 This is a flowchart of low-bit model training.
[0070] Figure 5 It is a schematic diagram of the processing steps of obtaining the maximum value of the feature map and then calculating the maximum value by the sliding average method.
[0071] Figure 6 This is a schematic diagram of the entire training process. DETAILED DESCRIPTION
[0072] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.
[0073] like Figure 1 As shown, the present invention relates to a quantization method for optimizing the activation of a neural network model, the method comprising the following steps:
[0074] S1: First train a full-precision model based on the dataset to achieve the target accuracy;
[0075] S2, based on the full-precision model training weights and activations quantized to 4 bits, and the maximum value of the feature map is counted during training;
[0076] S3, based on the 4-bit model trained in step S2, the weights and activations are quantized to a 2-bit model, and the maximum value of the feature map is recalculated.
[0077] Specifically, it includes the following:
[0078] 1 Full-precision model training:
[0079] 1) Training data:
[0080] The dataset for training the model is ImageNet1000, which is a subset of the ImageNet dataset, with approximately 1.2 million training sets, 50,000 validation sets, 150,000 test sets, and 1,000 categories.
[0081] 2) Model:
[0082] The basic neural network model used in this training is MobileNetV1, which is a model based on deep separable convolution. Some modifications mentioned in this article are made on this model.
[0083] 3) Select activation function:
[0084] In the MobileNetV1 model, BN and ReLU operations are added after each convolution layer, such as Figure 2 As shown in the figure, the MobileNetV1 model adds BN and Relu operations after each convolution layer. Conv2D is convolved first, then BN, and then Relu operation is performed. Among them, the BN batch normalization layer is defined as follows: when using batch gradient descent (or mini-batch), the output of the previous layer is normalized in the batch dimension, that is, in,
[0085] Where n is the input batch size, is the i-th output batch of the previous layer, and ε is a small number set to avoid division by 0.
[0086] However, since the trained model needs to be quantized to low bits, if the Relu activation function is used directly, the maximum value of the feature map output by each layer of activation is unknown and has no upper bound, which is not conducive to quantization to low bits. Since the feature map needs to be quantized to 2 bits later, the Relu activation function is replaced by ReluX during training:
[0087]
[0088] like Figure 3 As shown, Figure 2 The process of replacing the Relu activation function with ReluX during training.
[0089] 4) Training the network:
[0090] The basic steps for network training are: first use the Adam optimizer to train for 60 epochs, and then use the SGD optimizer until the training is completed.
[0091] 5) Test network performance:
[0092] Use the test set to test the network results.
[0093] 2. Low-bit model training:
[0094] like Figure 4 As shown in the figure, the low-bit model training process is as follows: first train the 4-bit model, then train the 2-bit model, then test the network effect, and finally output the network.
[0095] 1) Training a 4-bit model:
[0096] Data quantization: The data to be quantized is quantized according to the formula shown below to obtain low-bit data.
[0097]
[0098] Variable Description: W f For full precision data, it is an array. q For the simulated quantized data, max w Full precision data W f Maximum value, min w Full precision data W f The minimum value among them, b is the bit width after quantization.
[0099] In the first step of training, the weights and activations are quantized to 4 bits. While training the model, the maximum value of the feature map is obtained through the following processing and then the maximum value is calculated by the sliding average method. The processing steps are as follows:
[0100] 1: v = ReluX(v)
[0101] 2: channels=getChannelsNum(v)
[0102] 3: batchNum = getBatchNum(v)
[0103] 4: maxValue = 0.0
[0104] 5: for tag = 0 to bachNum do
[0105] 6: vValueTag = v[tag]
[0106] 7: perChannelMax=getChannelMax(vValueTag)
[0107] 8:
[0108] 9: end for
[0109] 10:
[0110] 11: max Value=max(max Value, 0.0)
[0111] 12: maxValue=min(maxValue, 3.0)
[0112] Function description: getChannelsNum(v) gets the number of channels of the feature map, getBatchNum(v) gets the size of the batch of the feature map, getChannelMax(v) gets the maximum value of each channel of the feature map, and reduceSum(v) calculates the sum of the variable v.
[0113] 3. The maxValue obtained by each layer of activation function is updated by the sliding average method. The formula is as follows:
[0114] v t =β·v t-1 +(1-β)·(θ t )
[0115] Variable description: v t is the value of variable v at time t, β is the weighting coefficient, θ t is the value of variable v at time t, v t-1 is the value of the variable v at time t-1.
[0116] The weight coefficient β is set to 0.996 for sliding average and 3.0 for initial value. In the training process, the value of X is maxValue and maxValue is recorded. In the inference process, the maximum value of statistics is used without recalculating the maximum value of feature. Figure 5As shown in the figure, during the training process, the value of X is maxValue, and maxValue is recorded. During the inference process, the statistical maximum value is used without recalculating the maximum value of the feature: After the BN operation, the maximum value computeMaxValue is calculated, and the function ReluX(x=maxValue) is calculated at the same time, where the parameter of the function is the maximum value maxValue. Figure 6 As shown in the figure, the entire training process is: calculate the weight weight, calculate quantization Quantize(weight) according to the weight, pre-layer PreLayer, and perform convolution and batch normalization according to the quantization result; further perform Relux(x=maxValue); and perform quantization Quantize(x).
[0117] 2) Training a 2-bit model
[0118] After the first step of training, a model with weights and activations quantized to 4 bits is obtained. Then, based on this model, a model with weights and activations quantized to 2 bits is trained. The specific process is consistent with the process of training a 4-bit model. When training a 2-bit model, maxValue needs to be re-counted. The value of the weight coefficient β is set to 0.998 during sliding average, and the initial value is set to 2.0.
[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A quantization method for optimizing activation of a neural network model, characterized in that: The method first trains a 4-bit model based on the full-precision model, and then trains a 2-bit model based on the 4-bit model. When training the 4-bit and 2-bit low-bit models, the maximum values of the feature maps are counted in a sliding average manner and are updated each time during training without relying on the statistical results of the previous model. The method comprises the following steps: S1, first train a full-precision model based on the data set to achieve the target accuracy; the step S1 further includes: S1.1, determine the training data: The dataset for training the model is ImageNet1000, which is a subset of the ImageNet dataset. It has 1.2 million training sets, 50,000 validation sets, 150,000 test sets, and 1,000 categories. S1.2, determine the training model: The basic neural network model used for training is MobileNetV1, which is a model based on deep separable convolution; S1.3, select activation function: In the MobileNetV1 model, batch normalization BN and activation function Relu operations are added after each layer of convolution; in step S1.3, since the trained model needs to be quantized to low bits and the feature map needs to be quantized to 2 bits afterwards, the activation function Relu operation can also replace the Relu activation function with ReluX during training, as shown in Formula 1: Formula 1; S1.4, training network: The basic steps for network training are: first use the adam optimizer to train for 60 epochs, and then use the SGD optimizer until the training is completed; S1.5, test network effect: Use the test set to test the network results; S2, training the model with weights and activations quantized to 4 bits based on the full-precision model, and counting the maximum value of the feature map during training; the step S2 further includes: S2.1, data quantization: quantize the data to be quantized according to formula 2 to obtain low-bit data. During training, quantize the weights and activations to 4 bits: Formula 2 Variable Description: For full precision data is an array, To simulate the quantized data, Full precision data The maximum value, Full precision data The minimum value in , b is the bit width after quantization; S2.2, while training the model, obtain the maximum value of the feature map and then use the sliding average method to calculate the maximum value, which is represented by maxValue; S2.3, the parameter maxValue obtained by the activation function of each layer is updated by the sliding average method, as shown in Formula 3: Formula 3 Variable Description: is the value of variable v at time t, β is the weighting coefficient, is the value of variable v at time t, is the value of variable v at time t-1; S3, based on the 4-bit model trained in step S2, weights and activations are trained to a 2-bit model, and the maximum value of the feature map is recalculated; step S3 includes: S3.1, data quantization: quantize the data to be quantized according to formula 2 to obtain low-bit data. During training, quantize the weights and activations to 2 bits: Formula 2 Variable Description: For full precision data is an array, To simulate the quantized data, Full precision data The maximum value, Full precision data The minimum value in , b is the bit width after quantization; S3.2, while training the model, obtain the maximum value of the feature map and then use the sliding average method to calculate the maximum value, which is represented by maxValue; S3.3, the parameter maxValue obtained by the activation function of each layer is updated by the sliding average method, as shown in Formula 3: Formula 3 Variable Description: is the value of variable v at time t, β is the weighting coefficient, is the value of variable v at time t, is the value of the variable v at time t-1.
2. A quantization method for optimizing activation of a neural network model according to claim 1, characterized in that: The step S2.2 obtains the maximum value by processing the following steps: 1: Assign ReluX(v) to parameter v: v=ReluX(v); 2: getChannelsNum(v) gets the number of channels of the feature map and assigns it to the parameter channels: channels=getChannelsNum(v); 3: getBatchNum(v) gets the size of the feature map batch and assigns it to the parameter batchNum: batchNum=getBatchNum(v); 4: Initialize parameter maxValue: maxValue=0.0; 5: When tag is from 0 to batchNum value, do: for tag=0 to batchNum do; 6: vValueTag=v[tag]; 7: getChannelMax(v) gets the maximum value of each channel of the feature map and assigns it to the parameter perChannelMax: perChannelMax=getChannelMax(vValueTag) 8: reduceSum(perChannelMax) calculates the sum of the variable perChannelMax and then divides it by the parameter channels value, adds it to the parameter maxValue value and assigns it to the parameter maxValue: maxValue+=reduceSum(perChannelMax) / channels; 9:end for; 10: The parameter maxValue value is divided by the batchNum value and assigned to the parameter maxValue: maxValue=maxValue / batchNum.
3. A quantization method for optimizing activation of a neural network model according to claim 2, characterized in that: In step S2.3, the weighting coefficient is the sliding average The value of is set to 0.996, and the initial value is set to 3.0, where the initial value is And t=0; then the value of X during training is , and record , the maximum value of statistics is used during inference without recalculating the maximum value of the feature.
4. A quantization method for optimizing activation of a neural network model according to claim 1, characterized in that: When training the 2-bit model, maxValue needs to be recalculated, and the weighted coefficient β is set to 0.998 during sliding average, and the initial value is set to 2.
0. In the initial value formula 3, And t=0.
5. A quantization method for optimizing activation of a neural network model according to claim 1, characterized in that: The method may further comprise: S4, test network effect; S5, output network.
Citation Information
Patent Citations
Method and device for adjusting artificial neural network
CN110555508A
4-bit quantization method and system of neural network
CN111882058A
