A deep learning quantization-aware training method and device, equipment and medium

Abstract

The application discloses a deep learning quantization perception training method and device, equipment and medium, and relates to the technical field of deep learning. The method comprises the following steps: loading a pre-trained deep learning network model, and solidifying weight parameters in the pre-trained deep learning network model to obtain a solidified network model; training preset trainable quantization factor parameters in the solidified network model to obtain a trained network model; loading the trained network model, and simultaneously training the trainable quantization factor parameters and the weight parameters in the trained network model to obtain a quantization perception trained model. The application can guarantee the accuracy of a deep learning network model quantized at a low bit, and improve the stability of quantization perception training of a lightweight network model and a pruned network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a deep learning quantization-aware training method and device, equipment and medium. BACKGROUND

[0002] With the rapid development of deep learning, the calculation amount of the model is getting larger and larger, and the inference time is getting longer and longer, which poses greater challenges to the inference deployment of deep learning, especially on some small computing hardware devices.

[0003] One of the current deep learning model acceleration methods is model quantization. Traditional model quantization methods include post-training static quantization (PQT) and quantization-aware training (QAT). The post-training static quantization method directly quantizes the trained model. Since quantization itself is a way of introducing noise, it can cause a significant decrease in the accuracy of models that are sensitive to noise. Most post-training static quantization uses 8-bit quantization, which greatly limits the application range of this method. The quantization-aware training method simulates the actual running process of quantization, including boundary clipping and rounding operations, during the training process, and provides the error generated during quantization as a feature to the network learning. In general, quantization-aware training can achieve better accuracy than post-training static quantization. However, since the gradient at almost every point in the rounding function is 0, the model cannot converge. The existing technical solution is to introduce STE (Straight Through Estimator) to directly return the weight gradient before quantization, so that the network model can be normally trained. However, this method causes the gradient information after quantization to be unable to be returned to the back propagation, resulting in a non-optimal solution. In addition, the current quantization-aware training method calculates the quantization coefficient by counting the weights and activations, which can cause the training to diverge when quantizing lightweight network models or pruned network models.

[0004] In summary, how to quantize deep learning network models is a problem that still needs to be further solved. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a deep learning quantization-aware training method, device, equipment and storage medium, which can ensure the accuracy of deep learning network models when quantized at low bit, and improve the stability of lightweight network models and pruned network models in quantization-aware training. The specific solutions are as follows:

[0006] In a first aspect, the present application discloses a deep learning quantization-aware training method, comprising:

[0007] loading a pre-trained deep learning network model, and solidifying weight parameters in the pre-trained deep learning network model to obtain a solidified network model;

[0008] training preset trainable quantization factor parameters in the solidified network model to obtain a trained network model;

[0009] loading the trained network model, and simultaneously training the trainable quantization factor parameters and the weight parameters in the trained network model to obtain a quantization-aware trained model.

[0010] Optionally, the trainable quantization factor parameters comprise a quantization factor s and a zero_point β.

[0011] Optionally, after the step of simultaneously training the trainable quantization factor parameters and the weight parameters in the trained network model to obtain a quantization-aware trained model, the method further comprises:

[0012] folding and merging constants of the quantization-aware trained model.

[0013] Optionally, the step of folding and merging constants of the quantization-aware trained model comprises:

[0014] loading the quantization-aware trained model, merging BN layers of the quantization-aware trained model to convolution layers, then merging the zero_point β to bias, further merging the zero_point β with the quantization factor s, and converting the weight parameters into integer form.

[0015] Optionally, the zero_point β is used to quantize floating-point data inputting the trainable quantization factor s into integer data.

[0016] Optionally, the step of loading a pre-trained deep learning network model comprises:

[0017] loading a pre-trained deep learning network model of float data type.

[0018] In a second aspect, the present application discloses a deep learning quantization-aware training device, comprising:

[0019] a first network model loading module configured to load a pre-trained deep learning network model;

[0020] a weight parameter solidification module, configured to solidify weight parameters in the pre-trained deep learning network model to obtain a solidified network model;

[0021] a quantization factor parameter training module, configured to train preset trainable quantization factor parameters in the solidified network model to obtain a trained network model;

[0022] a second network model loading module, configured to load the trained network model;

[0023] a quantization-aware training module, configured to simultaneously train the trainable quantization factor parameters and the weight parameters in the trained network model to obtain a quantization-aware trained model.

[0024] Optionally, the quantization-aware training module further comprises:

[0025] a constant folding and merging module, configured to perform constant folding and merging on the quantization-aware trained model.

[0026] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the deep learning quantization-aware training method as described above when executing a computer program saved in the memory.

[0027] In a fourth aspect, the present application discloses a computer readable storage medium, configured to store a computer program; wherein the computer program is executed by a processor to implement the deep learning quantization-aware training method as described above.

[0028] It can be seen that the pre-trained deep learning network model is loaded first, and the weight parameters in the pre-trained deep learning network model are solidified to obtain a solidified network model, then the preset trainable quantization factor parameters in the solidified network model are trained to obtain a trained network model, then the trained network model is loaded, and the trainable quantization factor parameters and the weight parameters in the trained network model are simultaneously trained to obtain a quantization-aware trained model. The present application can guarantee the precision of the deep learning network model in low bit quantization, and improve the stability of the quantization-aware training of the lightweight network model and the pruned network model. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0030] Figure 1 A deep learning quantization perception training method flow chart disclosed by the present application;

[0031] Figure 2 A deep learning quantization perception training device structure schematic diagram disclosed by the present application;

[0032] Figure 3 A structure diagram of an electronic device disclosed by the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0034] The embodiments of the present application disclose a deep learning quantization perception training method, referring to Figure 1 The method comprises the following steps:

[0035] Step S11: loading a pre-trained deep learning network model, and solidifying the weight parameters in the pre-trained deep learning network model to obtain a solidified network model.

[0036] In the embodiments, a pre-trained deep learning network model is loaded first, and then the weight parameters in the pre-trained deep learning network model are solidified to obtain a corresponding solidified network model. It should be noted that the pre-trained deep learning network model is specifically a trained pre-trained model of float data type, that is, a deep learning network model that has been trained for the first time and needs to be further optimized.

[0037] It can be understood that the quantization in the quantization perception training is a pseudo quantization. The data can be quantized from float32 to int8, that is, the floating point data is quantized to integer type data, and then the int8 is dequantized to float32. The specific quantization formula is as follows:

[0038]

[0039]

[0040] Wherein, s is a quantization factor, l and u correspond to the minimum and maximum values of x respectively, Q n and Q p correspond to the minimum and maximum values of the quantized values respectively, x qFor int8 data, round is the rounding function, and clip is the clipping function; wherein, the round(·) function is not derivable.

[0041] In this embodiment, g x represents the actual gradient in back propagation, and the specific calculation formula is as follows:

[0042]

[0043] wherein, g x is the partial derivative of the loss function with respect to the input x, is the partial derivative of the loss function with respect to x q , g x The specific derivation formula is as follows:

[0044]

[0045]

[0046]

[0047] wherein, is the second-order derivative, which can be approximated by the average value of the diagonal elements in the Hessian matrix in actual calculation; ∈ represents a very small number, such as represents the number of quantization bits.

[0048] Step S12: training the preset trainable quantization factor parameters in the solidified network model to obtain a trained network model.

[0049] In this embodiment, after the weight parameters in the pre-trained deep learning network model are solidified to obtain a solidified network model, in order to improve the convergence of the quantization model, the preset trainable quantization factor parameters in the above solidified network model can be further trained to obtain a corresponding trained network model, and the above trained network model is saved; wherein, the trainable quantization factor parameters specifically include a quantization factor s and a zero_point β, and the zero_point β is used to quantize floating-point data inputting the quantization factor s into integer data.

[0050] It should be pointed out that in this embodiment, the quantization factor s calculated by the weight is set as a trainable parameter, and in addition, the zero_point β is also set as a parameter for network model training.

[0051] Specifically, the derivation process of the quantization factor s gradient and the zero_point β is as follows: ​

[0052]

[0053]

[0054]

[0055] Step S13: loading the trained network model, and training the quantization factor parameters and the weight parameters in the trained network model simultaneously to obtain a quantization-aware trained model.

[0056] In this embodiment, after the trainable quantization factor parameters and the weight parameters in the trained network model are trained simultaneously to obtain a quantization-aware trained model, the trained network model is loaded, and the quantization factor parameters and the weight parameter gradients in the trained network model are trained simultaneously to further obtain a quantization-aware trained model.

[0057] Further, after the trainable quantization factor parameters and the weight parameters in the trained network model are trained simultaneously to obtain a quantization-aware trained model, constant folding and merging of the quantization-aware trained model can be further included. The specific constant folding and merging process can include loading the quantization-aware trained model, merging the BN layer of the quantization-aware trained model to the convolution layer, then merging the zero_point β to the bias, and further merging the quantization factor s, and converting the weight parameters into integer form. In this embodiment, the quantization-aware trained model needs to be further constant folded and merged. The BN (Batch Normalization) layer is merged to the convolution layer, and then the zero_point β is merged to the bias to improve the operation efficiency. The specific constant folding and merging process is as follows:

[0058] w q x q = (q w s w )(q x s x + β) = q w q x s w s x + βq w s w ;

[0059] wherein q w and q x are the integer form of the weight quantization and the integer form of the activation quantization, respectively; s w and s xquantization factor of weight and activation value respectively, β represents zero_point of input x, that is, the value corresponding to int data after quantization corresponding to float data 0 before quantization of input x.

[0060] Finally, in order to improve the efficiency of memory loading, the above quantization factor s is combined, and the weight parameter is converted into an integer form, and the specific formula is:

[0061] q act =S(q w q x -q w Z x +q bias )+Z act ;

[0062] Wherein, S represents s w *S x , Z x represents the ratio of zero_point of input x and quantization factor β / s x , Z act represents the zero_point of activation value output, q bias represents the quantized bias value.

[0063] It can be seen that the embodiment of the application loads a pre-trained deep learning network model first, and solidifies the weight parameters in the pre-trained deep learning network model to obtain a solidified network model, then trains the preset trainable quantization factor parameters in the solidified network model to obtain a trained network model, then loads the trained network model, and trains the trainable quantization factor parameters and the weight parameters in the trained network model at the same time to obtain a quantization-aware trained model. The embodiment of the application loads a pre-trained deep learning network model first, then solidifies the weight parameters in the model and trains the quantization parameters, so as to improve the convergence of the quantization model, and then trains the weight parameters and the quantization parameters at the same time to obtain a quantization-aware trained model. The embodiment of the application can ensure the precision of the deep learning network model in low bit quantization, improve the stability of the light network model and the pruned network model in quantization-aware training, and quantize the model to low bit, thereby reducing the hardware requirements of the deep learning deployment.

[0064] Correspondingly, the embodiment of the application also discloses a deep learning quantization-aware training device, as shown in Figure 2 , the device comprises:

[0065] The first network model loading module 11 is used for loading a pre-trained deep learning network model.

[0066] The weight parameter solidification module 12 is configured to solidify the weight parameters in the pre-trained deep learning network model to obtain a solidified network model.

[0067] The quantization factor parameter training module 13 is configured to train preset trainable quantization factor parameters in the solidified network model to obtain a trained network model.

[0068] The second network model loading module 14 is configured to load the trained network model.

[0069] The quantization-aware training module 15 is configured to simultaneously train the trainable quantization factor parameters and the weight parameters in the trained network model to obtain a quantization-aware trained model.

[0070] The specific working procedures of the above modules can refer to the corresponding content disclosed in the foregoing embodiments, and will not be described here.

[0071] As can be seen, in the embodiments of the present application, the pre-trained deep learning network model is loaded first, and the weight parameters in the pre-trained deep learning network model are solidified to obtain a solidified network model. Then, the preset trainable quantization factor parameters in the solidified network model are trained to obtain a trained network model. Then, the trained network model is loaded, and the trainable quantization factor parameters and the weight parameters in the trained network model are simultaneously trained to obtain a quantization-aware trained model. The embodiments of the present application can ensure the accuracy of the deep learning network model in low-bit quantization, and improve the stability of the light-weight network model and the quantization-aware training of the pruned network model.

[0072] In some specific embodiments, the trainable quantization factor parameters include a quantization factor s and a zero_point β.

[0073] In some specific embodiments, the quantization-aware training module 15 can further include:

[0074] The constant folding and merging module is configured to perform constant folding and merging on the quantization-aware trained model.

[0075] In some specific embodiments, the constant folding and merging module can specifically include:

[0076] The model loading unit is configured to load the quantization-aware trained model.

[0077] The merging and converting unit is configured to merge the BN layer of the quantization-aware trained model to the convolution layer, then merge the zero_point β to the bias, and further merge the quantization factor s, and convert the weight parameters into an integer form.

[0078] In some embodiments, the zero_point β is used to quantize the float-type data of the input and the trainable quantization factor s into integer-type data.

[0079] In some embodiments, the first network model loading module 11 can specifically include:

[0080] The network model loading unit is configured to load a pre-trained deep learning network model of a float data type.

[0081] Further, the embodiment of the present application further discloses an electronic device, Figure 3 The electronic device 20 is shown in the structure diagram according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0082] Figure 3 A structure diagram of an electronic device 20 is provided in the embodiment of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the deep learning quantization perception training method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0083] In the embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0084] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0085] The operating system 221 is configured to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of performing the deep learning quantization-aware training method disclosed in any of the preceding embodiments by the electronic device 20, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0086] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the deep learning quantization-aware training method disclosed in the preceding embodiments. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0087] The embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0088] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0089] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0090] Finally, it needs to be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0091] The deep learning quantization perception training method, device, equipment and medium provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples in this article. The above example is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A model deployment method for optimizing hardware device adaptability, characterized in that, The method comprises: loading a pre-trained deep learning network model and solidifying weight parameters in the pre-trained deep learning network model to obtain a solidified network model; the pre-trained deep learning network model is a network model of floating-point data type; training preset trainable quantization factor parameters in the solidified network model to obtain a trained network model; the trainable quantization factor parameters are configured with factor parameters for converting floating-point data type into integer data type; loading the trained network model and training the trainable quantization factor parameters and the weight parameters in the trained network model simultaneously to obtain a quantization-aware trained model; deploying the quantization-aware trained model to a target hardware device; wherein the hardware requirement of the quantization-aware trained model to the hardware device is lower than the hardware requirement of the pre-trained deep learning network model to the hardware device.

2. The model deployment method for optimizing hardware device adaptability according to claim 1, wherein, The trainable quantization factor parameters include a quantization factor s and a zero_point .

3. The model deployment method for optimizing hardware device adaptability according to claim 2, wherein, After the training of the trainable quantization factor parameters and the weight parameters in the trained network model simultaneously to obtain a quantization-aware trained model, the method further comprises: performing constant folding and merging on the quantization-aware trained model.

4. The model deployment method for optimizing hardware device adaptability according to claim 3, wherein, The constant folding and merging on the quantization-aware trained model comprises: loading the quantized perception trained model and merging the BN layer of the quantized perception trained model to the convolution layer, then merging the zero_point to the bias, and merging the quantization factor s, and converting the weight parameter to an integer form.

5. The model deployment method for optimizing hardware device adaptability according to any one of claims 2 to 4, characterized in that, the zero_point quantizing the floating-point data of the input into integer data using the trainable quantization factor s.

6. A model deployment apparatus for optimizing hardware device adaptability, characterized by, The method comprises: a first network model loading module for loading a pre-trained deep learning network model; the pre-trained deep learning network model is a network model of floating-point data type; a weight parameter solidification module for solidifying weight parameters in the pre-trained deep learning network model to obtain a solidified network model; a quantization factor parameter training module for training preset trainable quantization factor parameters in the solidified network model to obtain a trained network model; the trainable quantization factor parameters are configured with factor parameters for converting floating-point data type into integer data type; a second network model loading module for loading the trained network model; a quantization-aware training module for training the trainable quantization factor parameters and the weight parameters in the trained network model simultaneously to obtain a quantization-aware trained model; a deployment module for deploying the quantization-aware trained model to a target hardware device; wherein the hardware requirement of the quantization-aware trained model to the hardware device is lower than the hardware requirement of the pre-trained deep learning network model to the hardware device.

7. The model deployment apparatus for optimizing hardware device adaptability according to claim 6, wherein, The quantization-aware training module further comprises: a constant folding and merging module for performing constant folding and merging on the quantization-aware trained model.

8. An electronic device, comprising: The method comprises a processor and a memory; wherein the processor implements the model deployment method for optimizing hardware device adaptability as claimed in any one of claims 1 to 5 when executing the computer program saved in the memory.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the memory; wherein the computer program is executed by the processor to implement the model deployment method for optimizing hardware device adaptability as claimed in any one of claims 1 to 5.

Images