Deep learning floating point model training method and training device
By adjusting the model parameters to reduce outliers during the training process of deep learning floating-point models, the problem of many outliers in the parameter distribution of floating-point models is solved, which improves the efficiency and performance of model quantization and improves the accuracy of inference.
Patent Information
- Application Number
- CN202311075738.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-24
- Publication Date
- 2025-02-25
AI Technical Summary
In the existing model quantization technology, there are many outliers in the parameter distribution of floating-point models, resulting in poor quantization effect, especially when the parameter distribution is not suitable for quantization.
By obtaining the probability distribution of model parameters for each network layer during the training of the deep learning floating point model, the first loss function value between the desired normal distribution is determined based on the distribution, and the model parameters are adjusted to reduce the outliers until the expected value is reached.
Without changing the model structure, optimize the distribution of model parameters to make it more suitable for quantization, thereby improving the efficiency and performance of model quantization and improving the accuracy of reasoning.
Smart Images
Figure CN119513590A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular, to a training method and a training device for a deep learning floating-point model. Background Art
[0002] In the process of deep learning model deployment, model quantization is usually required to reduce memory usage and improve the running speed of the model. In the current model quantization scheme, whether it is post-training quantization (PTQ) or quantization-aware training (QAT), quantization is based on the trained floating-point model. Especially for PTQ, which is more commonly used in actual engineering, it does not retrain the model parameters after quantization, so its quantization effect depends on the parameter distribution of the floating-point model. If there are many outliers in the parameters, the quantization effect of the model cannot meet the requirements; and there is little research on how to make the parameter distribution of the floating-point model more suitable for quantization. Summary of the invention
[0003] The present invention provides a training method for a deep learning floating-point model to reduce the number of outlier parameters in the parameter distribution of the floating-point model.
[0004] A first aspect of an embodiment of the present application provides a training method for a deep learning floating-point model, the method comprising:
[0005] Input the sample data into the deep learning floating-point model to be trained.
[0006] Get the probability distribution of model parameters for each network layer in the deep learning floating-point model to be trained.
[0007] For each network layer, based on the model parameter probability distribution of the network layer, determine a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution,
[0008] Determine the second loss function value between the output result of the deep learning floating-point model to be trained and the set target output result,
[0009] According to the sum of the first loss function values and the second loss function values, the model parameters of the deep learning floating-point model to be trained are adjusted so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced.
[0010] Repeat this process until the deep learning floating-point model to be trained meets expectations.
[0011] A second aspect of the present application is a training device for a deep learning floating-point model, the device comprising:
[0012] The statistics module is used to obtain the probability distribution of model parameters of each network layer in the deep learning floating-point model to be trained.
[0013] A first loss function value determination module is used to determine, for each network layer, a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution based on the model parameter probability distribution of the network layer.
[0014] A second loss function value determination module is used to determine a second loss function value between an output result of the deep learning floating-point model to be trained and a set target output result.
[0015] The adjustment module is used to adjust the model parameters of the deep learning floating-point model to be trained according to the sum of the first loss function values and the second loss function values, so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced, and the adjustment is repeated until the deep learning floating-point model to be trained meets expectations.
[0016] A third aspect of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the training methods for the deep learning floating-point model are implemented.
[0017] The training method of the deep learning floating-point model provided in the present application reduces the outliers in the probability distribution of the model parameters of each network layer to be trained by introducing a first loss function value between the probability distribution of the model parameters of each network layer and the expected normal distribution, and makes the model parameters more suitable for the quantization distribution without changing the model structure, thereby improving the efficiency and performance of the model quantization while using fewer resources, thereby improving the accuracy of reasoning during the model application process. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A schematic diagram of floating-point quantization.
[0019] Figure 2 A flowchart of a method for training a deep learning floating-point model according to an embodiment of the present application.
[0020] Figure 3 A schematic diagram of the probability distribution curves of different kurtosis coefficients.
[0021] Figure 4 A schematic diagram of the probability distribution curves of different skewness coefficients.
[0022] Figure 5 A schematic diagram of the deep learning multi-classification model structure of this embodiment.
[0023] Figure 6Training for this example Figure 5 A flowchart of the deep learning multi-classification model.
[0024] Figure 7 This is a schematic diagram of the weight distribution of the first layer of convolution of the backbone when the floating-point model is quantized after training without using the kurtosis coefficient as the loss function (left figure) and with using the kurtosis coefficient as the loss function (right figure) in this embodiment.
[0025] Figure 8 A schematic diagram of a deep learning floating-point model training device according to an embodiment of the present application.
[0026] Fig. 9 Another schematic diagram of the deep learning floating-point model training device according to an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.
[0028] The applicant found that PTQ quantization does not require training. It only needs to provide some sample images, and then calibrate on the trained model to calculate the required quantization parameters. Therefore, it requires the parameter distribution of the model to be suitable for quantization in order to obtain good quantization results, that is, the accuracy loss of the model is small. Figure 1 As shown, Figure 1 This is a schematic diagram of a symmetrical quantization of floating point quantization. Taking the int8 quantization of the floating point model as an example, int8 quantization is to convert the floating point number x f Mapped to 8-bit representation x in the range [-128, 127] by scaling factor scale q , that is, x q =clip(round(x f / scale)
[0029] Among them, round means rounding to integers, and clip means truncating outliers to the range of [-128,127]. For the scale value, it is obtained as follows:
[0030] amax=max(abs(x f ))
[0031] scale=(2*amax) / (127-(-128))=(2*amax) / 255
[0032] Among them, abs means taking the absolute value. Since symmetric quantization is adopted, assuming amax>abs(amin), then amin=-amax, amax-amin=2*amax.
[0033] It can be seen that the distribution of floating-point parameters is crucial to quantization. If amax is an outlier, some quantized values cannot be fully used to express the model parameters, reducing the representation range of int8 data, resulting in the performance of the quantized model being worse than that of the floating-point model. Although commonly used methods such as asymmetric quantization, mean square error, and cross entropy can select relatively appropriate extreme values (amax, amin), their essence is the statistics of model parameters and cannot change the parameter distribution of the model.
[0034] Since PTQ does not train the quantization model parameters or only fine-tunes the quantization model parameters, when there are many outliers in the floating-point model parameters, the quantization effect is poor and cannot meet engineering applications. Therefore, the effect of PTQ quantization is very dependent on the parameter distribution of the floating-point model.
[0035] In view of this, an embodiment of the present application provides a training method for a deep learning floating-point model, which makes the distribution of model parameters conform to the normal distribution during the training process of the floating-point model to reduce outliers.
[0036] See also Figure 2 As shown, Figure 2 A flow chart of a training method for a deep learning floating-point model according to an embodiment of the present application. The training method includes:
[0037] Step 201: input sample data into the deep learning floating point model to be trained.
[0038] Among them, deep learning floating-point model refers to a model in which the model parameters are floating-point numbers;
[0039] It should be understood that the deep learning floating-point model can be a model for any task, including but not limited to image target detection tasks, text recognition tasks, speech recognition tasks, etc.
[0040] Step 202, obtaining the probability distribution of model parameters of each network layer in the deep learning floating point model to be trained,
[0041] Among them, the network layer refers to the network layer that includes model parameters, such as convolutional layer, fully connected layer, and normalization layer; since Relu and Softmax are activation functions, they have no parameters, and the global pooling layer (GAP, Global Average Pooling) also has no parameters, parameter statistics are not performed, and the batch sample normalization layer (BatchNorm) will be integrated into the convolutional layer during the quantization process.
[0042] Thus, the probability distribution of model parameters of each network layer of the deep learning floating-point model to be trained can be obtained.
[0043] As an example, the weight probability distribution of each convolutional layer in the deep learning floating-point model to be trained is obtained, the model parameters of the normalized layer are fused to the convolutional layer connected to the normalized layer, and the weight probability distribution of each fused convolutional layer in the deep learning floating-point model to be trained is obtained.
[0044] Get the weight probability distribution of each fully connected layer in the deep learning floating-point model to be trained.
[0045] There are two cases for the fusion of convolutional layer and normalization layer: one is that the convolutional layer is in front and the normalization layer is in the back, and the other is that the convolutional layer is in the back and the normalization layer is in the front. The fusion methods of the two cases are as follows.
[0046] Assume that the calculation formula of the convolution layer is: X′=WX+b, where W is the weight in the convolution layer, b is the bias in the convolution layer, X is the input image feature data of the convolution layer, and X′ is the output image feature data of the convolution layer.
[0047] The calculation formula of the normalization layer is: Among them, Y is the output image feature data of the normalization layer, X′ is the input image feature data of the normalization layer, and γ, u, σ, ε, and β are all normalization layer parameters.
[0048] When the convolution layer comes first and the normalization layer comes later, we have:
[0049]
[0050] Therefore,
[0051] The fused convolutional layer weight W′ is:
[0052] The convolutional layer bias b′ after fusion:
[0053] When the normalization layer is in front and the convolution layer is in the back, we have:
[0054]
[0055] Therefore,
[0056] The fused convolutional layer weight W′:
[0057] The convolutional layer bias b′ after fusion:
[0058] It should be understood that the fusion of normalization to the convolution layer is only a formal fusion to facilitate the distribution of parameters actually used in the statistical model quantification operation process. It does not really fuse the normalization layer and convolution into one operator, so the model network structure is not changed. Since the fusion of the normalization layer to the convolution layer does not change the distribution of the convolution layer parameters, it is sufficient to obtain the weight probability distribution of each fused convolution layer in the deep learning floating-point model to be trained.
[0059] Step 203, for each network layer, based on the model parameter probability distribution of the network layer, determine a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution, wherein the loss function value characterizes the degree to which the model parameter probability distribution deviates from the normal distribution.
[0060] As an example, the first loss function value is calculated using the kurtosis coefficient and / or the skewness coefficient of the probability distribution.
[0061] Thus, the first loss function value of each network layer of the deep learning floating-point model to be trained can be obtained.
[0062] Step 204: determine a second loss function value between the output result of the deep learning floating-point model to be trained and the set target output result, where the loss function value represents the degree to which the output result of the deep learning floating-point model to be trained deviates from the target output result.
[0063] Step 205: adjusting the model parameters of the deep learning floating-point model to be trained according to the sum of the first loss function values and the second loss function values, so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced.
[0064] Steps 201 to 205 are repeatedly performed until the deep learning floating-point model to be trained meets expectations.
[0065] The embodiments of the present application can optimize the distribution of model parameters without changing the floating-point model structure, so that the model parameters are suitable for quantization, and are beneficial to improving the accuracy of quantization, thereby improving the accuracy of deep learning model reasoning. For example, the accuracy of the detection results of the deep learning model used for image target detection can be improved. The training method of the embodiments of the present application can not only be used for PTQ, but also as the initial value of QAT.
[0066] To facilitate the understanding of the present application, the following is explained in combination with a deep learning model for multi-classification as an example. It should be understood that the present application is not limited to deep learning models for multi-classification, and deep learning models for other tasks can be applied, including but not limited to deep learning models for image target detection, deep learning models for image processing, for example, deep learning models for image reconstruction, deep learning models for image super-resolution conversion, etc.
[0067] Taking the image detection model as an example, in the training process of the image detection floating-point model: after the image is input into the target detection model, the first step is to calculate the classification loss and position regression loss of the target detection task respectively; the second step is to fuse the BN and convolution layers of the image detection floating-point model. The fusion here is only a formal fusion to facilitate the statistical distribution of weights. Then, the kurtosis coefficient of the weights of the fused image detection floating-point model is calculated layer by layer, and its absolute value is used as the loss to adjust the shape of the weight distribution so that it is close to the normal distribution while not becoming too smooth, which will affect the expression ability of the model; the losses calculated in the first and second steps will jointly optimize the target detection task. Among them, the second step optimizes the weight distribution of the image detection floating-point model, limits the outliers of the weights of each layer of the image detection floating-point model, and makes its distribution closer to the normal distribution to facilitate the subsequent model quantization.
[0068] After the training of the image detection floating-point model is completed, the PTQ quantization operation is performed. Specifically, the image detection floating-point model structure is converted into the model structure in the quantized reasoning process, that is, the model is optimized at the structural level, such as the fusion of BN and convolution, and unnecessary operations (excluding parameters, such as deformation operators) are eliminated; then, the calibration set image is used to calibrate the floating-point model with optimized structure. The main operations are: first, the distribution of statistical weights to find the maximum and minimum values of weight quantization; second, the maximum and minimum values of the activation output (that is, the value output by each layer of the model) during the statistical model reasoning process; finally, the model is quantized using the found extreme values of weights and activations.
[0069] From the PTQ quantization process of the floating-point model for image detection, it can be seen that reducing the outliers of the weights can obtain a more compact distribution, thereby making full use of the expressive power of fixed-point values (such as int8). At the same time, reducing the outliers of the weights is also conducive to reducing the outliers of the activations. Therefore, the method proposed by the present invention is conducive to improving the PTQ quantization accuracy of the model.
[0070] The image to be detected is input into the quantization model obtained after quantization by the image detection floating-point model PTQ. Since the model structure is optimized and the quantization accuracy is improved, the model structure can perform image processing processes more quickly, such as feature extraction, feature quantization, etc., which means that the image processing speed is improved.
[0071] The basic concepts of kurtosis and skewness are explained below.
[0072] The kurtosis coefficient is a statistical feature that characterizes the height of the peak of the probability distribution curve at the mean value, that is, it is a statistic that describes the steepness of the distribution of all values in the population. This statistic needs to be compared with the normal distribution. Intuitively, the kurtosis coefficient reflects the sharpness of the peak in the probability distribution curve.
[0073] See also Figure 3 As shown, Figure 3 It is a schematic diagram of probability distribution curves with different kurtosis coefficients. Compared with the normal distribution, the kurtosis coefficient can describe the steepness of the distribution of data near the mean.
[0074] If the kurtosis coefficient is greater than 0, it indicates that the shape of the peak is sharper, which is steeper than the peak of the normal distribution and has a peak shape, which means that the increase in variance is caused by low-frequency extreme differences that are greater than or less than the mean value;
[0075] If the kurtosis coefficient is less than 0, it is flatter than the peak of the normal distribution and has a low peak shape.
[0076] The larger the absolute value of the kurtosis coefficient, the greater the difference between the steepness of its distribution shape and the normal distribution.
[0077] Based on the basic concept of kurtosis, by appropriately limiting the kurtosis, the distribution of values can be closer to normal distribution to reduce outliers, which is exactly what the quantitative model needs. However, if the kurtosis is too low, the parameter distribution will tend to be uniform, which is not conducive to the expressiveness of the model.
[0078] The kurtosis coefficient can be calculated as follows: the ratio of the fourth-order central moment of the model parameter to the variance of the model parameter, minus a value of 3. Mathematically expressed as:
[0079]
[0080] in, represents the average value of the data, Used to calculate the variance of the data, n is used to calculate the fourth central moment of the data.
[0081] The skewness coefficient is similar to the kurtosis coefficient. It is also a statistic that describes the shape of data distribution. It describes the characteristic statistic of the symmetry of data distribution. This statistic also needs to be compared with the normal distribution.
[0082] See also Figure 4 As shown, Figure 4 A schematic diagram of the probability distribution curves of different skewness coefficients.
[0083] Skewness ≈ 0 means that the data distribution has the same degree of skewness as the normal distribution;
[0084] Skewness>0 means that the data distribution is positively skewed (right-skewed) compared with the normal distribution, that is, there is a long tail on the right, there are more extreme values on the right end of the data, and the degree of dispersion on the right side of the data mean is strong;
[0085] Skewness < 0 means that the data distribution is negatively skewed (left-skewed) compared with the normal distribution, that is, there is a long tail on the left, there are more extreme values on the left end of the data, and the degree of dispersion on the left side of the data mean is strong.
[0086] The larger the absolute value of the skewness, the greater the skewness of the distribution.
[0087] The calculation method of the skewness coefficient can be as follows: the average value of the cube of the ratio of the mean deviation of the model parameters to the standard deviation of the model parameters. It can be expressed mathematically as:
[0088]
[0089] in, represents the data mean, and σ represents the data standard deviation.
[0090] Based on the basic concept of skewness coefficient, by appropriately limiting the skewness coefficient, the distribution of values can be made closer to a normal distribution to reduce outliers, which is exactly what a quantitative model needs.
[0091] See Figure 5. Figure 5 It is a schematic diagram of the structure of the deep learning multi-classification model of this embodiment. The input sample data is 128*128*3, and the output is the score of two types of targets. The main parameters of the model include two convolutional layers Conv_1, Conv_2, and a fully connected layer (FC). The batch sample normalization (BN) layer also has training parameters, which are generally fused to the convolutional layer before quantization. The present invention performs parameter distribution statistics on the convolutional layer fused with BN, so the convolution kernel and the normalization layer can be supervised and trained using the kurtosis coefficient or the skewness coefficient at the same time, but the network structure is not changed; Relu and Softmax are activation functions, which have no parameters; GAP is a 2D global pooling layer, which also has no parameters.
[0092] See also Figure 6 As shown, Figure 6 Training for this example Figure 5 A flow chart of the deep learning multi-classification model. The training method includes:
[0093] Step 601: input the sample data into the deep learning multi-classification model to be trained, so that the deep learning multi-classification model to be trained processes the sample data.
[0094] Step 602, calculate the kurtosis coefficient of the weights in the convolution layer Conv_1, and obtain a first loss function value 1 for characterizing the kurtosis coefficient difference.
[0095] As an example, the number of input channels of the convolution layer Conv_1 is 3, the number of output channels is 64, and the convolution kernel size is 3*3, so the number of weights is 1728. Among them, the BN1 layer has been fused to the convolution layer Conv_1.
[0096] First, based on the number of weights and all weights in the convolutional layer Conv_1, the weight mean of the convolutional layer Conv_1 is calculated, which can be expressed mathematically as:
[0097]
[0098] Among them, w 1i is the i-th weight in the convolutional layer Conv_1, is the weight mean of the convolutional layer Conv_1,
[0099] Then, according to the kurtosis coefficient calculation formula, the absolute value of the kurtosis coefficient of the convolution layer Conv_1 is calculated to obtain the first loss function value of the convolution layer Conv_1 used to characterize the kurtosis coefficient difference, which is recorded as the first loss function value 1 and expressed in mathematical formula as follows:
[0100]
[0101] in, is the first loss function value 1 used to characterize the kurtosis coefficient difference.
[0102] Taking the absolute value of the kurtosis coefficient as the loss function value of the weight is equivalent to the role of the regularized L1 and L2 norms, both of which affect the distribution of the weights. Taking the absolute value of the weight kurtosis coefficient can make the kurtosis coefficient approach 0, ensuring that the distribution approaches the normal distribution, while maintaining a certain degree of diversity and not weakening the expressive power of the model.
[0103] As another example, the absolute value of the skewness coefficient of the convolution layer Conv_1 can be calculated according to the skewness coefficient calculation formula to obtain the first loss function value of the convolution layer Conv_1 used to characterize the skewness coefficient difference, which is recorded as the first loss function value 4 and expressed in mathematical formula as follows:
[0104]
[0105] in, is the first loss function value 4 used to characterize the difference in skewness coefficient, and σ1 is the weight standard deviation of the convolution layer Conv_1.
[0106] Step 603, calculate the kurtosis coefficient of the weight in the convolution layer Conv_2, and obtain the first loss function value 2 for characterizing the kurtosis coefficient difference.
[0107] As an example, the number of input channels of the convolution layer Conv_2 is 64, the number of output channels is 64, and the convolution kernel size is 3*3, so the number of weights is 36864. Among them, the BN2 layer has been fused to the convolution layer Conv_2.
[0108] First, based on the number of weights and all weights in the convolutional layer Conv_2, the weight mean of the convolutional layer Conv_2 is calculated, which can be expressed mathematically as:
[0109]
[0110] Among them, w 2i is the i-th weight in the convolutional layer Conv_2, is the weight mean of the convolutional layer Conv_2,
[0111] Then, according to the kurtosis coefficient calculation formula, the absolute value of the kurtosis coefficient of the convolution layer Conv_2 is calculated to obtain the first loss function value of the convolution layer Conv_2 used to characterize the kurtosis coefficient difference, which is recorded as the first loss function value 2 and expressed mathematically as follows:
[0112]
[0113] in, is the first loss function 2 used to characterize the difference in kurtosis coefficient,
[0114] The first loss function value is calculated for the parameters in the model according to the layers. In practice, the model contains multiple layers, so the first loss function value contains multiple items. The first loss function 2 described here is actually the calculation result of the convolution layer 2 in the first loss function. The following are similar.
[0115] As another example, the absolute value of the skewness coefficient of the convolution layer Conv_2 can be calculated according to the skewness coefficient calculation formula to obtain the first loss function value of the convolution layer Conv_2 used to characterize the skewness coefficient difference, which is recorded as the first loss function value 5 and expressed in mathematical formula as follows:
[0116]
[0117] in, is the first loss function value 5 used to characterize the difference in skewness coefficient, and σ2 is the weight standard deviation of the convolution layer Conv_2.
[0118] Step 604, calculate the kurtosis coefficient of the weights in the fully connected layer to obtain a first loss function value 3 for characterizing the kurtosis coefficient difference.
[0119] As an example, the number of weights in the fully connected layer is 128.
[0120] Then, based on the number of weights and all weights in the fully connected layer, the mean weight in the fully connected layer is calculated, which can be expressed mathematically as follows:
[0121]
[0122] Among them, w 3i is the i-th weight in the fully connected layer, is the weight mean of the fully connected layer,
[0123] Then, according to the kurtosis coefficient calculation formula, the absolute value of the kurtosis coefficient of the fully connected layer is calculated to obtain the first loss function value of the fully connected layer used to characterize the kurtosis coefficient difference, which is recorded as the first loss function value 3 and expressed mathematically as follows:
[0124]
[0125] in, is the first loss function 3 used to characterize the difference in kurtosis coefficient,
[0126] As another example, the absolute value of the skewness coefficient of the fully connected layer can be calculated according to the skewness coefficient calculation formula to obtain the first loss function value of the fully connected layer used to characterize the skewness coefficient difference, which is recorded as the first loss function value 6 and expressed in mathematical formula as follows:
[0127]
[0128] in, is the first loss function value 6 used to characterize the difference in skewness coefficient, and σ3 is the weight standard deviation of the fully connected layer.
[0129] Step 605, calculate the total loss function value of the model, and adjust the model parameters according to the total loss function value, and repeatedly perform steps 601 to 605 until the model to be trained meets the expectations.
[0130] As an example, the total loss function value is the final loss function value of the model, which includes two parts:
[0131] One part is the classification loss function value of the model, that is, the second loss function value between the classification result and the target classification result,
[0132] The other part is the kurtosis coefficient of the model parameters, which is the sum of all the first loss function values used to characterize the differences in kurtosis coefficients.
[0133] Furthermore, the sum of all first loss function values used to characterize the kurtosis coefficient difference is weighted using the first weighting coefficient η, wherein the weighting coefficient η is set according to the actual situation so that the sum of all first loss function values used to characterize the kurtosis coefficient difference is less than the second loss function value, so as to ensure the accuracy of the deep learning task. It can be expressed as follows in mathematical formula:
[0134]
[0135] Among them, Loss is the total loss function value, and Loss_clc is the second loss function value.
[0136] As another example, the total loss function value is the final loss function value of the model, which includes two parts:
[0137] One part is the classification loss function value of the model, that is, the second loss function value between the classification result and the target classification result,
[0138] The other part is the skewness coefficient of the model parameters, which is the sum of all the first loss function values used to characterize the differences in skewness coefficients.
[0139] Furthermore, the second weighting coefficient α is used to weight the sum of all first loss function values used to characterize the skewness coefficient difference. The second weighting coefficient α is set according to the actual situation so that the sum of all first loss function values used to characterize the skewness coefficient difference is less than the second loss function value to ensure the accuracy of the deep learning task. It can be expressed as follows:
[0140]
[0141] Among them, Loss is the total loss function value, and Loss_clc is the second loss function value.
[0142] As another example, the total loss function value is the final loss function value of the model, including three parts:
[0143] The first part is the classification loss function value of the model, that is, the second loss function value between the classification result and the target classification result.
[0144] The second part is the kurtosis coefficient of the model parameters, which is the sum of all the first loss function values used to characterize the differences in kurtosis coefficients.
[0145] The third part is the skewness coefficient of the model parameters, which is the sum of all the first loss function values used to characterize the differences in skewness coefficients.
[0146] Furthermore, the first weighting coefficient η is used to weight the sum of all first loss function values for characterizing the difference in kurtosis coefficients, and the second weighting coefficient α is used to weight the sum of all first loss function values for characterizing the difference in skewness coefficients. The first weighting coefficient η and the second weighting coefficient α are set according to the actual situation so that the sum of all first loss function values is less than the second loss function value to ensure the accuracy of the deep learning task. It can be expressed as follows:
[0147]
[0148] Among them, Loss is the total loss function value, and Loss_clc is the second loss function value.
[0149] This embodiment adds the model parameter distribution to the training process of the floating-point model without changing the model structure, thereby solving the problem that the model parameters cannot be effectively trained during the PTQ quantization process; using the kurtosis coefficient and / or the skewness coefficient as the loss function of the model parameter distribution can make the trained floating-point model parameter distribution more consistent with the normal distribution, reduce the outliers in the parameter distribution, and thus make the floating-point model more consistent with the quantization requirements; using the absolute value of the kurtosis coefficient as the loss function value can prevent the model parameters from becoming flat in the process of tending to the normal distribution, thereby ensuring the expressiveness of the model.
[0150] This application takes the detection model as an example to illustrate the use of kurtosis as a loss function to train a floating-point model and then test the quantization accuracy of its PTQ. The detection model is yolox, which consists of three parts: backbone, neck, and head. The backbone is mobilenetv2_1.0, the data set is the VOC data set, and the weight of the kurtosis is 0.01.
[0151] The specific implementation process is as follows: First, the YOLOX floating-point model with added kurtosis coefficient loss is trained using the VOC training set. The training framework used here is mmdet. After training, the floating-point model is exported as an onnx file and converted into an mnn file using the conversion tool provided by the MNN framework. Then, 1,000 images are randomly selected from the training set and the test set as the calibration set.
[0152] The following table compares the PTQ quantization effect when the kurtosis coefficient is used as the loss function and when the kurtosis coefficient is not used as the loss function. The quantization form of the model is W8A8 symmetric quantization, that is, both weights and activations are represented in int8. As can be seen from the table, using the kurtosis coefficient as the loss function achieves better quantization effect while ensuring detection accuracy.
[0153] Model Parameter quantity (M) Floating point precision PTQ quantification accuracy yolox 2.312 77.60 77.40 Yolox + Kurtosis supervision 2.312 77.55 77.50
[0154] See also Figure 7 As shown, Figure 7 This is a schematic diagram of the weight distribution of the first layer of convolution of the backbone when quantization is performed after training without using the kurtosis coefficient as the loss function (left figure) and with the kurtosis coefficient as the loss function (right figure). The outliers in the left figure are around 1.25, and the outliers in the right figure are around 0.75. Generally speaking, the first layer of convolution of the detection model is difficult to quantize. By analyzing the first layer of convolution of the backbone of the model when the kurtosis coefficient is not used as the loss function and when the kurtosis coefficient is used as the loss function, it can be seen that using the kurtosis coefficient as the loss function limits the outliers to a certain extent, so it achieves a better quantization effect.
[0155] See also Figure 8 As shown, Figure 8 This is a schematic diagram of a deep learning floating-point model training device according to an embodiment of the present application. The device includes:
[0156] The statistics module is used to obtain the probability distribution of model parameters of each network layer in the deep learning floating-point model to be trained.
[0157] A first loss function value determination module is used to determine, for each network layer, a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution based on the model parameter probability distribution of the network layer.
[0158] A second loss function value determination module is used to determine a second loss function value between an output result of the deep learning floating-point model to be trained and a set target output result.
[0159] The adjustment module is used to adjust the model parameters of the deep learning floating-point model to be trained according to the sum of the first loss function values and the second loss function values, so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced, and the adjustment is repeated until the deep learning floating-point model to be trained meets expectations.
[0160] As an example, the first loss function value determining module includes:
[0161] The first calculation submodule is used to calculate the absolute value of the kurtosis coefficient of the probability distribution of the model parameters as the first loss function value for characterizing the kurtosis coefficient difference,
[0162] The second calculation submodule is used to calculate the absolute value of the skewness coefficient of the probability distribution of the model parameters as a first loss function value for characterizing the skewness coefficient difference.
[0163] See also Fig. 9 As shown, Fig. 9Another schematic diagram of a deep learning floating-point model training device according to an embodiment of the present application. The training device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the deep learning floating-point model training method according to the embodiment of the present application.
[0164] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0165] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0166] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the training method of the deep learning floating-point model described in the embodiment of the present application are implemented.
[0167] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0168] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A training method for a deep learning floating-point model, characterized in that: The method includes: Input the sample data into the deep learning floating-point model to be trained. Get the probability distribution of model parameters for each network layer in the deep learning floating-point model to be trained. For each network layer, based on the model parameter probability distribution of the network layer, determine a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution, Determine the second loss function value between the output result of the deep learning floating-point model to be trained and the set target output result, According to the sum of the first loss function values and the second loss function values, the model parameters of the deep learning floating-point model to be trained are adjusted so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced. Repeat this process until the deep learning floating-point model to be trained meets expectations.
2. The training method according to claim 1, characterized in that: The first loss function value is determined according to at least one of a kurtosis coefficient and a skewness coefficient of a probability distribution of a model parameter.
3. The training method according to claim 2, characterized in that: In a case where the first loss function value is determined according to the kurtosis coefficient, the determining of the first loss function value for describing the probability distribution of the model parameter and the expected normal distribution comprises: Calculate the average value of the model parameters of the network layer, Based on the average value, the absolute value of the kurtosis coefficient and / or the absolute value of the skewness coefficient of the probability distribution of the model parameters is calculated to obtain a first loss function value for characterizing the difference in kurtosis coefficient and / or a first loss function value for characterizing the difference in skewness coefficient.
4. The training method according to claim 2, characterized in that: The step of adjusting the parameters of the deep learning floating-point model to be trained according to the sum of the first loss function values and the second loss function values includes: Accumulating the first loss function values for characterizing the difference in kurtosis coefficients and / or the first loss function values for characterizing the difference in skewness coefficients, The accumulated first loss function value is weighted so that the accumulated first loss function value is less than the second loss function value, The weighted accumulated first loss function value and the second loss function value are added to obtain the total loss function value. According to the total loss function value, adjust the model parameters of the deep learning floating-point model to be trained.
5. The training method according to claim 4, characterized in that: The obtaining of the probability distribution of model parameters of each network layer in the deep learning floating-point model to be trained includes: Get the weight probability distribution of each convolutional layer in the deep learning floating-point model to be trained; The model parameters of the normalization layer are fused to the convolutional layer connected to the normalization layer. Get the weight probability distribution of each fused convolutional layer in the deep learning floating-point model to be trained, Get the weight probability distribution of each fully connected layer in the deep learning floating-point model to be trained.
6. The training method according to claim 5, characterized in that: The first loss function value of the convolutional layer is calculated as follows: For any convolutional layer, According to the number of input channels, the number of output channels, and the size of the convolution kernel of the convolution layer, the number of weights of the convolution layer is determined. The number of weights is the product of the number of input channels, the number of output channels, and the size of the convolution kernel. Accumulate all weights of the convolutional layer, According to the accumulated weights and the number of weights of the convolutional layer, the average value of the weights of the convolutional layer is calculated. Using the average value, the kurtosis coefficient and / or the skewness coefficient are calculated to obtain the first loss function value of the convolution layer.
7. The training method according to claim 5, characterized in that: The first loss function value of the fully connected layer is calculated as follows: For any fully connected layer, Accumulate all weights of the fully connected layer, According to the accumulated weights and the number of weights of the fully connected layer, the average value of the weights of the fully connected layer is calculated. Using the average value, the kurtosis coefficient and / or the skewness coefficient are calculated to obtain the first loss function value of the fully connected layer.
8. The training method according to claim 1, characterized in that: The deep learning floating-point model is a model for image target detection, or a model for image multi-classification, or a model for image processing.
9. A training device for a deep learning floating-point model, characterized in that: The device includes: The statistics module is used to obtain the probability distribution of model parameters of each network layer in the deep learning floating-point model to be trained. A first loss function value determination module is used to determine, for each network layer, a first loss function value used to describe the difference between the model parameter probability distribution and the expected normal distribution based on the model parameter probability distribution of the network layer. A second loss function value determination module is used to determine a second loss function value between an output result of the deep learning floating-point model to be trained and a set target output result. The adjustment module is used to adjust the model parameters of the deep learning floating-point model to be trained according to the sum of the first loss function values and the second loss function values, so that the outliers in the probability distribution of the model parameters of each network layer of the deep learning floating-point model to be trained are reduced, and the adjustment is repeated until the deep learning floating-point model to be trained meets expectations.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the training method for a deep learning floating-point model as described in any one of claims 1 to 8.
Citation Information
Cited By
Baseline evaluation method for air conditioner load under various meteorological conditions
CN119903458A