Cross-range deep neural network parameter hybrid quantization method
By using a cross-range hybrid quantization method in deep neural networks, the activation value and weights are quantized respectively, which solves the problem of degradation of model performance after quantization in the existing technology, and achieves a more stable and generalized deep neural network model.
Patent Information
- Application Number
- CN202411996345.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-05-02
AI Technical Summary
In the prior art, deep neural network parameter quantization methods will lead to a significant decline in model performance, especially when quantizing low bit widths, which will increase the error of training and inference results, affecting the stability and generalization ability of the model.
The hybrid quantization method of deep neural network parameters across the range is used to quantify the activation value through the PACT quantization method. After the neural network model training is completed, the weight value is quantified using the optimized LSQ quantization method, and the numerical values exceeding the quantization range are filtered and additional quantized, and their information is retained.
It realizes that without significantly increasing the computing and storage overhead, retaining more information, improving the stability and generalization capabilities of the model, and improving the performance under low-precision quantization.
Smart Images
Figure CN119918589A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of quantization technology, and specifically relates to a cross-range hybrid quantization method for deep neural network parameters. Background Art
[0002] Parameter quantization in neural networks refers to converting floating-point parameters (such as weights and activation values) in neural networks into lower-precision values (such as integers). The core goal of this technology is to improve the application performance of deep learning models in resource-constrained environments such as low-power, embedded devices, and mobile devices by reducing the storage and computational overhead of the model. Neural network models usually contain a large number of parameters, which are stored as 32-bit floating-point numbers (FP32), resulting in very high memory usage and computational overhead.
[0003] Although parameter quantization technology has made significant progress, there are still some problems and challenges. The performance of the quantized model may drop significantly, especially when low-bit width quantization is used (such as compressing weights from 32-bit floating point numbers to 8-bit integers). Because the precision of floating point numbers is lost during the quantization process, some details are lost, which may increase the error of training and inference results and affect the final accuracy; the stability of the model may be affected, especially in some specific layers or specific types of neural networks, low-precision quantization will cause unstable network training or the model cannot generalize well; the quantization process has a greater impact on activation functions, especially nonlinear activation functions (such as ReLU, Sigmoid, Tanh, etc.). The quantized activation value may distort the output of the activation function, thereby affecting the training and inference performance of the network. Summary of the invention
[0004] In view of this, the present invention provides a cross-range hybrid quantization method for deep neural network parameters to solve the problem that the quantization method in the prior art will significantly reduce the performance of the model.
[0005] The technical solution adopted by the present invention is as follows:
[0006] Hybrid quantization methods for parameters of deep neural networks across a range of domains, including:
[0007] Step 1: During the training of the deep neural network model, the deep neural network activation is quantified using the PACT quantization method;
[0008] Described step 1 specifically comprises the following steps:
[0009] Step 1.1: During the training of the neural network model, the parameter C of the clipping function in the PACT quantization method is learned as shown in the following formula:
[0010] a *=clip(a,-C,C)
[0011] In the formula, a is the activation value of a layer in the neural network model, C is the parameter of the clipping function, which is expressed as the threshold parameter of the activation value quantization, controlling the range of the activation value - C represents the lower limit, C represents the upper limit, and a * is the activation value after clipping;
[0012] The clip function formula is as follows:
[0013]
[0014] in, and are the left and right thresholds of the fixed-point number distribution range after the activation value parameter is quantized, round is the rounding function, and θ is a constant greater than 1. Values greater than C and less than -C that exceed the quantization range are enlarged by θ times, and the quantized values are closer to the true values, thus obtaining the effect of cross-range quantization.
[0015] Step 1.2: Quantize the activation value as shown below:
[0016]
[0017] Where min and max are the minimum and maximum activation values after clipping, and round indicates rounding operation.
[0018] Step 2: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range during the quantization process are screened and quantized, retaining the information of these values.
[0019] The step 2 specifically includes:
[0020] (1) After the neural network model training is completed, the weight parameters of the LSQ quantization model are initialized based on the weight parameters of the pre-trained full-precision model, and the quantization step size in the LSQ quantization function is initialized based on the weights of the full-precision model and the statistical information of the input feature map;
[0021] The quantization step size in the LSQ quantization function is initialized according to the weight of the full-precision model and the statistical information of the input feature map by the following formula:
[0022]
[0023] Among them, the || || function is used to calculate the L1 norm of the input matrix, and the weights of the convolutional layer are w, c out is the number of convolution kernels, c in is the number of channels of the convolution kernel, h kand w k are the length and width of the convolution kernel respectively, and s is the step size.
[0024] (2) In the process of weight quantization, the numbers whose values exceed the quantization range are divided into two parts and quantized separately;
[0025] The process of dividing the number exceeding the quantization range into two parts and quantizing them separately in the weight quantization process specifically includes:
[0026] Step A1: Let x be a number that exceeds the quantization range, Q be the quantization range threshold corresponding to the number x, then x can be split into two parts, the formula is as follows:
[0027] x=Q+(xQ)
[0028] Step A2: When cross-range quantization is performed, let the quantization step size be s. For the original floating-point number w, first divide it by the quantization step size s and use the round function to map it to a fixed-point number.
[0029] For floating-point numbers that exceed the quantization threshold, they are first split into the corresponding quantization threshold and the remaining part, and then the remaining part is additionally quantized. Finally, the fixed-point number obtained in the previous step is multiplied by the quantization step size s to achieve quantization. The specific formula of the final output function is as follows:
[0030]
[0031] Among them, round is the rounding function, w is the original floating point number, s is the step size, and are the left threshold and right threshold of the fixed-point number distribution range after the weight parameter is quantized, and θ is a constant greater than 1.
[0032] Compared with the direct truncation quantization function, cross-range quantization splits the values that exceed the quantization range into two parts. Although it will increase the storage and calculation amount, the additional calculation amount is less because the values that exceed the quantization range themselves account for a small proportion.
[0033] (3) Using the gradient rule of the LSQ quantization function, the gradient descent algorithm is dynamically optimized during the training process to update the model parameters and step size s.
[0034] The gradient rule of the LSQ quantization function is used to dynamically optimize the gradient descent algorithm during the training process, and the updating of the model parameters and the step size s is realized by the following formula:
[0035]
[0036] where 1(·) is an indicator function used to calculate the gradient only when x does not exceed the quantization range.
[0037] A cross-range hybrid quantization system for deep neural network parameters, comprising:
[0038] Activation quantization module: During the training of the deep neural network model, the deep neural network activation is quantified using the PACT quantization method;
[0039] Weight quantization module: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range in the two quantization processes are screened and quantized, retaining the information of these values.
[0040] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0041] 1. In the present invention, different quantization methods are used for activation values and weights, respectively, and the effects of different output layers in the network are fully considered.
[0042] 2. In the present invention, the weight quantization range is expanded from within the threshold to outside the threshold. Compared with directly cutting off the values outside the threshold, the cross-range quantization is achieved and more information outside the quantization range is retained.
[0043] 3. In the present invention, unlike the traditional one-time activation quantization, activation quantization is performed after the feature extraction layer to retain more input features. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The present invention will now be described by way of example with reference to the accompanying drawings, in which:
[0045] Figure 1 This is the full flow chart of neural network parameter quantization of the present invention.
[0046] Figure 2 This is the activation value quantization flow chart of the present invention.
[0047] Figure 3 This is a weight quantization flow chart of the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0049] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0052] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0054] Example 1
[0055] like Figure 1-Figure 3 As shown, an embodiment of the present invention discloses a cross-range deep neural network parameter hybrid quantization method, including:
[0056] Step 1: During the training of the deep neural network model, the deep neural network activation is quantified using the PACT quantization method;
[0057] Described step 1 specifically comprises the following steps:
[0058] Step 1.1: During the training of the neural network model, the parameter C of the clipping function in the PACT quantization method is learned as shown in the following formula:
[0059] a * =clip(a,-C,C)
[0060] In the formula, a is the activation value of a layer in the neural network model, C is the parameter of the clipping function, which is expressed as the threshold parameter of the activation value quantization, controlling the range of the activation value - C represents the lower limit, C represents the upper limit, and a * is the activation value after clipping;
[0061] The clip function formula is as follows:
[0062]
[0063] in, and are the left and right thresholds of the fixed-point number distribution range after the activation value parameter is quantized, round is the rounding function, and θ is a constant greater than 1. Values greater than C and less than -C that exceed the quantization range are enlarged by θ times, and the quantized values are closer to the true values, thus obtaining the effect of cross-range quantization.
[0064] Step 1.2: Quantize the activation value as shown below:
[0065]
[0066] Where min and max are the minimum and maximum activation values after clipping, and round indicates rounding operation.
[0067] Step 2: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range during the quantization process are screened and quantized, retaining the information of these values.
[0068] The step 2 specifically includes:
[0069] (1) After the neural network model training is completed, the weight parameters of the LSQ quantization model are initialized based on the weight parameters of the pre-trained full-precision model, and the quantization step size in the LSQ quantization function is initialized based on the weights of the full-precision model and the statistical information of the input feature map;
[0070] The quantization step size in the LSQ quantization function is initialized according to the weight of the full-precision model and the statistical information of the input feature map by the following formula:
[0071]
[0072] Among them, the || || function is used to calculate the L1 norm of the input matrix, and the weights of the convolutional layer are w, c out is the number of convolution kernels, c in is the number of channels of the convolution kernel, h k and w k are the length and width of the convolution kernel respectively, and s is the step size.
[0073] (2) In the process of weight quantization, the numbers whose values exceed the quantization range are divided into two parts and quantized separately;
[0074] The process of dividing the number exceeding the quantization range into two parts and quantizing them separately in the weight quantization process specifically includes:
[0075] Step A1: Let x be a number that exceeds the quantization range, Q be the quantization range threshold corresponding to the number x, then x can be split into two parts, the formula is as follows:
[0076] x=Q+(xQ)
[0077] Step A2: When cross-range quantization is performed, let the quantization step size be s. For the original floating-point number w, first divide it by the quantization step size s and use the round function to map it to a fixed-point number.
[0078] For floating-point numbers that exceed the quantization threshold, they are first split into the corresponding quantization threshold and the remaining part, and then the remaining part is additionally quantized. Finally, the fixed-point number obtained in the previous step is multiplied by the quantization step size s to achieve quantization. The specific formula of the final output function is as follows:
[0079]
[0080] Compared with the direct truncation quantization function, cross-range quantization splits the values that exceed the quantization range into two parts. Although it will increase the storage and calculation amount, the additional calculation amount is less because the values that exceed the quantization range themselves account for a small proportion.
[0081] (3) Using the gradient rule of the LSQ quantization function, the gradient descent algorithm is dynamically optimized during the training process to update the model parameters and step size s.
[0082] The gradient rule of the LSQ quantization function is used to dynamically optimize the gradient descent algorithm during the training process, and the updating of the model parameters and the step size s is realized by the following formula:
[0083]
[0084] where 1(·) is an indicator function used to calculate the gradient only when x does not exceed the quantization range.
[0085] Example 2
[0086] This embodiment proposes a cross-range deep neural network parameter hybrid quantization system, including:
[0087] Activation quantization module: During the training of the deep neural network model, the deep neural network activation is quantified using the PACT quantization method;
[0088] Weight quantization module: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range in the two quantization processes are screened and quantized, retaining the information of these values.
[0089] Example 3
[0090] This embodiment proposes a specific implementation method based on Embodiment 1.
[0091] The embodiment conducts image classification and recognition experiments on CIFAR-10, CIFAR-100 and ImageNet datasets to verify the effectiveness of the cross-range quantization method. The CIFAR-10 dataset contains 50,000 training images and 10,000 verification images, with a total of 10 categories. The CIFAR-100 dataset also contains 50,000 training images and 10,000 verification images, with a total of 100 categories. The ImageNet dataset contains 50,000 training images and 50,000 verification images, with a total of 1,000 categories.
[0092] A ResNet-18 model was selected to train a full-precision model, and then a low-bit model was fine-tuned on the trained full-precision model. For training the full-precision model, SGD was used as the optimizer, the momentum was set to 0.9, the weight decay was 0.0001, the initial learning rate was 0.1, and a total of 100 epochs were trained. The learning rate was multiplied by 0.1 at the 25th and 75th epochs, respectively, and the sample batch size was 128. The detailed comparison data is shown in the following table:
[0093] Table 1 Image classification accuracy in ImageNet dataset (%)
[0094] Quantization Bits 2 3 4 LSQ 66.38 69.01 70.30 Quantification across ranges 68.54 70.34 71.02
[0095] Table 2 Image classification accuracy in CIFAR-10 dataset (%)
[0096] Quantization Bits 2 3 4 LSQ 90.13 91.32 91.76 Quantification across ranges 91.24 91.76 91.91
[0097] Table 3 Image classification accuracy in CIFAR-100 dataset (%)
[0098] Quantization Bits 2 3 4 LSQ 66.68 67.92 68.23 Quantification across ranges 67.54 68.34 68.62
[0099] Experimental results show that the cross-range quantization method is better than the LSQ quantization method under different quantization bits.
[0100] The circuits, electronic components and modules involved are all prior art and can be fully implemented by those skilled in the art. Needless to say, the content protected by the present invention does not involve improvements to software and methods.
[0101] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0102] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cross-range hybrid quantization method for deep neural network parameters, characterized in that: include: Step 1: During the training of the deep neural network model, the activation of the deep neural network is quantified using the PACT quantization method; Step 2: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range during the quantization process are screened and quantized, retaining the information of these values.
2. The cross-range deep neural network parameter hybrid quantization method according to claim 1, characterized in that: Described step 1 specifically comprises the following steps: Step 1.1: In the process of training the neural network model, first take a part of the samples from the training set for pre-training. During the pre-training process, the parameter C of the clipping function in the PACT quantization method is learned, as shown in the following formula: a * =clip(a,-C,C) In the formula, a is the activation value of the feature layer in the neural network model, C is the parameter of the clipping function, which is expressed as the threshold parameter of the activation value quantization, controlling the range of the activation value - C represents the lower limit, C represents the upper limit, and a * is the activation value after clipping; The clip function formula is as follows: in, and are the left and right thresholds of the fixed-point number distribution range after the activation value parameter is quantized, round is the rounding function, θ is a constant greater than 1, and the values greater than C and less than -C that exceed the quantization range are enlarged by θ times. The quantized value is closer to the true value, and the effect of cross-range quantization is obtained. Step 1.2: Quantize the activation value, as shown in the following formula: Where min and max are the minimum and maximum activation values after clipping, and round indicates rounding operation.
3. The cross-range deep neural network parameter hybrid quantization method according to claim 1, characterized in that: The step 2 specifically includes: After the neural network model training is completed, the weight parameters of the LSQ quantization model are initialized based on the weight parameters of the pre-trained full-precision model, and the quantization step size in the LSQ quantization function is initialized based on the weights of the full-precision model and the statistical information of the input feature map; In the weight quantization process, the numbers whose values exceed the quantization range are divided into two parts and quantized separately; By using the gradient rule of the LSQ quantization function, the gradient descent algorithm is dynamically optimized during the training process to update the neural network model parameters and step size s.
4. The cross-range deep neural network parameter hybrid quantization method according to claim 3, characterized in that: The quantization step size in the LSQ quantization function is initialized according to the weight of the full-precision model and the statistical information of the input feature map by the following formula: Among them, the || || function is used to calculate the L1 norm of the input matrix, and the weights of the convolutional layer are w, c out is the number of convolution kernels, c in is the number of channels of the convolution kernel, h k and w k are the length and width of the convolution kernel respectively, and s is the step size.
5. The cross-range deep neural network parameter hybrid quantization method according to claim 4, characterized in that: The process of dividing the number exceeding the quantization range into two parts and quantizing them separately in the weight quantization process specifically includes: Step A1: Let x be a number that exceeds the quantization range, Q be the quantization range threshold corresponding to the number x, then x can be split into two parts, the formula is as follows: x=Q+(xQ) Step A2: When cross-range quantization is performed, let the quantization step size be s. For the original floating-point number w, first divide it by the quantization step size s and use the round function to map it to a fixed-point number. For floating-point numbers that exceed the quantization threshold, they are first split into the corresponding quantization threshold and the remaining part, and then the remaining part is additionally quantized. Finally, the fixed-point number obtained in the previous step is multiplied by the quantization step size s to achieve quantization. The specific formula of the final output function is as follows: Among them, round is the rounding function, w is the original floating point number, s is the step size, and are the left threshold and right threshold of the fixed-point number distribution range after the weight parameter is quantized, and θ is a constant greater than 1.
6. The cross-range deep neural network parameter hybrid quantization method according to claim 5, characterized in that: The gradient rule of the LSQ quantization function is used to dynamically optimize the gradient descent algorithm during the training process, and the weight parameters and step size s are updated by the following formula: where 1(·) is an indicator function used to calculate the gradient only when x does not exceed the quantization range, represents the derivative of y with respect to x, q max is the upper limit of the quantization threshold.
7. A cross-range deep neural network parameter hybrid quantization system, used to implement a cross-range deep neural network parameter hybrid quantization method according to claims 1-6, characterized in that: include: Activation quantization module: During the training of the deep neural network model, the deep neural network activation is quantified using the PACT quantization method; Weight quantization module: After the neural network model training is completed, the weights of the neural network model are quantized using the optimized LSQ quantization method, and the values that exceed the quantization range in the two quantization processes are screened and quantized, retaining the information of these values.