Neural network model quantification method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides a quantification method and device of a neural network model, equipment, a medium and a program product. The method comprises the following steps: according to a performance index of a to-be-deployed chip, quantifying a neural network model to obtain a first quantified neural network model; performing first training on the neural network model after the first quantization; the training frequency corresponding to the first training is greater than or equal to 3; in the process of the first training, according to loss change values of at least two adjacent training, the quantization digit of at least one layer of the neural network model after the first quantization is adjusted, and a neural network model after the second quantization is obtained; taking the second quantized neural network model as a first quantized neural network model to continue first training, and iteratively executing the above steps until the first training is finished to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used for being deployed on a to-be-deployed chip.

Description

Technical Field

[0001] The present application relates to the field of model compression technology, and specifically to a quantization method and device, equipment, medium, and program product for a neural network model. Background Art

[0002] Low-bit quantization technology is an important means of compressing deep learning models. As the scale of neural network models continues to expand, the demand for computing resources and storage space has also increased significantly. Traditional 32-bit or 16-bit floating-point representations not only take up a lot of storage space, but also have high computational complexity and require high computing power from electronic devices. Summary of the invention

[0003] The present application provides a quantization method and apparatus, device, medium, and program product for a neural network model, which is beneficial to reducing the computing power and memory requirements of the chip to be deployed.

[0004] In a first aspect, an embodiment of the present application provides a method for quantizing a neural network model, the method comprising: quantizing the neural network model according to performance indicators of a chip to be deployed to obtain a first quantized neural network model; performing a first training on the first quantized neural network model; wherein the number of training times corresponding to the first training is greater than or equal to 3; during the first training, adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to loss change values ​​of at least two adjacent trainings to obtain a second quantized neural network model; continuing the first training with the second quantized neural network model as the first quantized neural network model, and iteratively executing the above steps until the first training is completed to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed implements the function corresponding to the neural network model based on the dynamically quantized neural network model.

[0005] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the neural network model is quantized according to the performance indicators of the chip to be deployed, and then in the first training process, according to the loss change value obtained by training, the quantization bit number of the layer of the first quantized neural network model is dynamically adjusted to obtain a dynamically quantized neural network model, and the dynamically quantized neural network model is used to be deployed on the chip to be deployed; in this way, through the performance indicators and dynamic quantization of the chip to be deployed, a dynamically quantized neural network model with fewer parameters and lower computational complexity is obtained, and based on the loss change value, the quantization bit number of the layer of the first quantized neural network model is dynamically adjusted, which is beneficial to avoid the problems of accuracy loss and training instability of the first quantized neural network model, and reduce the computing power and memory requirements of the chip to be deployed.

[0006] In some embodiments, the adjusting the number of quantization bits of at least one layer of the first quantized neural network model also includes: during the first training process, according to the accuracy change value between the accuracy of the first quantized neural network model obtained after training and the accuracy of the neural network model, adjusting the number of quantization bits of at least one layer of the first quantized neural network model.

[0007] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, during the first training process, the quantization bit number of at least one layer of the first quantized neural network model is adjusted according to the accuracy change value of the first quantized neural network model obtained by training and the accuracy change value of the neural network model. In this way, the quantization position of each layer is dynamically adjusted according to the loss change value and the accuracy change value obtained by training, thereby further ensuring the performance of the dynamically quantized neural network model while reducing the computational complexity of the model, thereby further reducing the computing power and memory requirements of the chip to be deployed.

[0008] In some embodiments, the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the accuracy change value of the accuracy of the first quantized neural network model obtained by training and the accuracy of the neural network model, including: when the accuracy change value is greater than or equal to the accuracy threshold, according to the layer characteristics of the layer of the first quantized neural network model obtained by current training, obtaining the Hessian matrix corresponding to the layer; wherein the layer characteristics include: the eigenvalue of the layer, the gradient of the layer; when the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits; when the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the second number of quantization bits; the second number of quantization bits is greater than the first number of quantization bits.

[0009] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, when the accuracy change value is greater than or equal to the accuracy threshold, the Hessian matrix corresponding to the layer is obtained according to the characteristic value of the layer of the first quantized neural network model obtained by the current training and the gradient of the layer; the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the modulus of the Hessian matrix. In this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding layer to quantization, the number of quantization bits of the layer is adjusted according to the value of the Hessian matrix, that is, the number of quantization bits of the layer is adjusted according to the sensitivity of the layer to quantization; thus, for the quantization-sensitive layer, a larger number of quantization bits is set for it; for the relatively insensitive layer, a smaller number of quantization bits is set for it; that is, through the hybrid quantization method, the quantized neural network model is balanced in terms of accuracy and running speed, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0010] In some embodiments, the quantization bit number of at least one layer of the first quantized neural network model is adjusted according to the loss change value of at least two adjacent trainings, including: when the loss change value is less than or equal to the loss threshold, according to the layer characteristics of the layer of the first quantized neural network model obtained from the latter training of two adjacent trainings, obtaining the Hessian matrix corresponding to the layer; wherein the layer characteristics include: the eigenvalue of the layer, the gradient of the layer; when the modulus of the Hessian matrix is ​​less than the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the first quantization bit number; when the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the second quantization bit number; the second quantization bit number is greater than the first quantization bit number.

[0011] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, when the loss change value is less than or equal to the loss threshold, the Hessian matrix corresponding to the layer is obtained according to the characteristic value of the layer of the first quantized neural network model obtained in the latter training of two adjacent trainings and the gradient of the layer; according to the modulus of the Hessian matrix, the quantization bit number of at least one layer of the first quantized neural network model is adjusted. In this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding layer to quantization, the quantization bit number of the layer is adjusted according to the value of the Hessian matrix, that is, the quantization bit number of the layer is adjusted according to the sensitivity of the layer to quantization; thus, for the quantization-sensitive layer, a larger quantization bit number is set for it; for the relatively insensitive layer, a smaller quantization bit number is set for it; that is, through the hybrid quantization method, the quantized neural network model is balanced in terms of accuracy and running speed, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0012] In some embodiments, the layer characteristics also include: the computational complexity of the layer, the precision of the layer; when the loss change value is less than or equal to the loss threshold, the method also includes: performing statistical analysis on the data distribution characteristics of the layer of the first quantized neural network model obtained from the latter training of two adjacent trainings to obtain the size range of the eigenvalues ​​of the layer; adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to at least one of the following: the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer.

[0013] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, not only the first quantized neural network model is dynamically quantized based on the Hessian matrix corresponding to the layer, but also the first quantized neural network model is dynamically quantized based on the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer. In this way, since the computational complexity of the layer directly affects the computing power requirements of the quantized neural network model for the chip to be deployed, the precision of the layer and the size range of the eigenvalues ​​of the layer directly affect the performance of the quantized neural network model, the first quantized neural network model is dynamically quantized based on the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer, which is beneficial to further ensure the performance of the dynamically quantized neural network model while reducing the computational complexity of the model, thereby further reducing the computing power and memory requirements of the chip to be deployed.

[0014] In some embodiments, the method of obtaining the first quantized neural network model further includes: performing a second training on the neural network model; wherein the number of training times of the second training is greater than or equal to 1; identifying the category of the layer of the neural network model; wherein the category of the layer includes: convolutional layer, fully connected layer; obtaining the Hessian matrix corresponding to the convolutional layer according to the layer characteristics of the convolutional layer obtained by the second training; wherein the layer characteristics include: the eigenvalue of the convolutional layer, the gradient of the convolutional layer; according to the modulus of the Hessian matrix, adjusting the number of quantization bits of at least one of the convolutional layers, and adjusting the number of quantization bits of at least one of the fully connected layers, to obtain the first quantized neural network model.

[0015] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the neural network model is subjected to a second training; the convolution layer and the fully connected layer of the neural network model are identified; the Hessian matrix corresponding to the convolution layer is obtained according to the eigenvalue of the convolution layer and the gradient of the convolution layer obtained by the second training; the number of quantization bits of at least one convolution layer is adjusted according to the modulus of the Hessian matrix, and the number of quantization bits of at least one fully connected layer is adjusted to obtain the first quantized neural network model; in this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding convolution layer to quantization, the number of quantization bits of the convolution layer is adjusted according to the value of the Hessian matrix, that is, the number of quantization bits of the convolution layer is adjusted according to the sensitivity of the convolution layer to quantization, and the fully connected layer is a layer that is relatively insensitive to quantization relative to the convolution layer, so the number of quantization bits of the fully connected layer is uniformly adjusted; thereby achieving a balance between the accuracy and running speed of the quantized neural network model, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0016] In some embodiments, the category of the layer also includes: an activation layer; after obtaining the first quantized neural network model, the method also includes: performing statistical analysis on the data distribution characteristics of the activation layer obtained by the second training to obtain the size range of the eigenvalues ​​of the activation layer; when the size range of the eigenvalues ​​of the activation layer is greater than a range threshold, adjusting the activation function corresponding to the activation layer in the first quantized neural network model to a truncated activation function to obtain an optimized neural network model; and using the optimized neural network model as the first quantized neural network model.

[0017] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the data distribution characteristics of the activation layer obtained by the second training are statistically analyzed to obtain the size range of the eigenvalues ​​of the activation layer; when the size range of the eigenvalues ​​of the activation layer is greater than the range threshold, the activation function corresponding to the activation layer in the first quantized neural network model is adjusted to a truncated activation function to obtain an optimized neural network model; the optimized neural network model is used as the first quantized neural network model. In this way, when the size range of the eigenvalues ​​of the activation layer is greater than the range threshold, the maximum and minimum values ​​of the eigenvalues ​​are limited by the truncated activation function, and the size range of the eigenvalues ​​is reduced, so that in the process of dynamic quantization, the loss of the eigenvalues ​​after dynamic quantization is less, thereby improving the stability of the neural network model after dynamic quantization; and because the size range of the eigenvalues ​​is reduced, the number of quantization bits required is also less, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0018] In a second aspect, an embodiment of the present application provides a quantization device for a neural network model, the device comprising: a pre-quantization module, configured to quantize the neural network model according to the performance indicators of a chip to be deployed, to obtain a first quantized neural network model; a first training module, configured to perform a first training on the first quantized neural network model; wherein the number of training times corresponding to the first training is greater than or equal to 3; a dynamic quantization module, configured to adjust the number of quantization bits of at least one layer of the first quantized neural network model according to the loss change values ​​of at least two adjacent trainings during the first training, to obtain a second quantized neural network model; the second quantized neural network model is used as the first quantized neural network model to continue the first training, and the above steps are iteratively performed until the first training is completed to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed implements the function corresponding to the neural network model based on the dynamically quantized neural network model.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the quantization method described in the first aspect is implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the quantization method described in the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the quantization method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the implementation process of a quantization method of a neural network model provided in an embodiment of the present application Figure 1 ;

[0023] Figure 2 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 1 ;

[0024] Figure 3 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 2 ;

[0025] Figure 4 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 3 ;

[0026] Figure 5 A schematic diagram of the implementation process of a quantization method of a neural network model provided in an embodiment of the present application Figure 2 ;

[0027] Figure 6 A schematic diagram of a quantization device for a neural network model provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments\other embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments\other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] In a related technology, a model quantization method, device, storage medium and electronic device are provided. The method includes: quantizing the model to be quantized according to the first candidate parameter value of the quantization parameter of the model to be quantized to obtain the first candidate quantized model; determining the second candidate parameter value of the quantization parameter of the model to be quantized by a preset parameter determination strategy according to the first candidate parameter value, quantizing the model to be quantized according to the second candidate parameter value to obtain the second candidate quantized model; selecting a reference quantized model from the first candidate quantized model and the second candidate quantized model according to the performance values ​​of the first candidate quantized model and the second candidate quantized model, determining the parameter value corresponding to the reference quantized model as the first candidate parameter value, using the reference quantized model as the first candidate quantized model, returning to execute the corresponding steps until the preset stop condition is met, and obtaining the quantized model corresponding to the model to be quantized.

[0035] In another related technology, a model quantization method, device, equipment and storage medium are provided, which relates to the field of deep learning model technology. The method includes: obtaining the statistical value of the activation value of the batch normalization (BN) layer in the model to be processed during the training process, and determining the current processing structure layer in the model to be processed in turn; determining the quantization threshold matching the current processing structure layer according to the type and statistical value of the current processing structure layer; determining the quantization conversion coefficient according to the quantization threshold, and quantizing the activation value of the original data type of the current processing structure layer to the activation value of the target data type according to the quantization conversion coefficient to obtain the quantized model. The technical solution of the embodiment of the present application can be used to solve the problem that the prior art needs to provide a calibration data set, infer the calibration data set with single-precision floating point (FP32) accuracy, and obtain the activation value of each layer, resulting in a long overall calibration time and affecting the model quantization accuracy.

[0036] In another related technology, a model compression method, device and readable storage medium are provided. The method includes: obtaining a first model, the first model is a floating-point model trained using a training set; scaling the weights of each convolutional layer in the first model to obtain a second model; quantizing the second model based on a preset quantization mode to obtain a third model; the third model is a fixed-point model; determining a target convolutional layer in the third model, and correcting the number of shifts of the target convolutional layer to obtain a target model; the target convolutional layer refers to a convolutional layer in the third model whose number of shifts does not meet a predetermined range; the number of shifts of each convolutional layer in the target model meets the predetermined range. The embodiment of the present application can save the hardware resources required for the fixed-point model obtained after quantization when performing convolution calculations.

[0037] However, since the quantization methods of related technologies use 8-bit or lower representation, the defects of traditional low-bit quantization methods are as follows: (1) Precision loss: Under low-bit representation, the inference accuracy of the model tends to drop significantly, especially in tasks that require highly precise calculations. (2) Training instability: Nonlinear mapping and truncation errors in the quantization process can cause the training process to be unstable or even unable to converge. (3) Hardware adaptation difficulties: Different hardware platforms have different levels of support for low-bit quantization, which makes adaptation challenges in practical applications.

[0038] In view of this, the present application provides a quantization method for a neural network model. Figure 1 A schematic diagram of the implementation process of a quantization method of a neural network model provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes steps 101 to 103:

[0039] Step 101, quantizing the neural network model according to the performance index of the chip to be deployed, to obtain a first quantized neural network model;

[0040] Step 102, performing a first training on the first quantized neural network model; the number of training times corresponding to the first training is greater than or equal to 3;

[0041] Step 103, during the first training process, according to the loss change value of at least two adjacent trainings, the quantization bit number of at least one layer of the first quantized neural network model is adjusted to obtain a second quantized neural network model; the second quantized neural network model is used as the first quantized neural network model to continue the first training, and the above steps are iteratively executed until the first training is completed to obtain a dynamically quantized neural network model; the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed can realize the corresponding function of the neural network model based on the dynamically quantized neural network model.

[0042] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the neural network model is quantized according to the performance index of the chip to be deployed, and then in the first training process, according to the loss change value obtained by training, the quantization bit number of the layer of the first quantized neural network model is dynamically adjusted to obtain a dynamically quantized neural network model, and the dynamically quantized neural network model is used to be deployed on the chip to be deployed. In this way, through the performance index and dynamic quantization of the chip to be deployed, a dynamically quantized neural network model with fewer parameters and lower computational complexity is obtained, and based on the loss change value, the quantization bit number of the layer of the first quantized neural network model is dynamically adjusted, which is beneficial to avoid the problems of accuracy loss and training instability of the first quantized neural network model, and reduce the computing power and memory requirements of the chip to be deployed.

[0043] The following describes further optional implementations and related terms of each of the above steps.

[0044] In step 101, the neural network model is quantized according to the performance index of the chip to be deployed to obtain a first quantized neural network model.

[0045] It should be understood that in the embodiments of the present application, there is no limitation on the neural network model, and the neural network model can be any neural network model, for example: a neural network model applied to the image field, a neural network model applied to the security field, a neural network model applied to the aerospace field, a neural network model applied to the intelligent driving field, etc.

[0046] In some embodiments, the performance indicators include at least one of the following: the speed at which the chip to be deployed runs the neural network model, the memory size of the chip to be deployed, the central processing unit (CPU) occupancy rate when the chip to be deployed runs the neural network model, the graphics processing unit (GPU) occupancy rate when the chip to be deployed runs the neural network model, the memory occupancy rate when the chip to be deployed runs the neural network model, the power consumption when the chip to be deployed runs the neural network model, the temperature when the chip to be deployed runs the neural network model, and the instruction set when the chip to be deployed runs the neural network model; at least one of the following instruction sets includes: the instruction set corresponding to the assembly language supported by the chip to be deployed, the instruction set of the Compute Unified Device Architecture (CUDA) supported by the chip to be deployed, and the instruction set corresponding to the code library supported by the chip to be deployed.

[0047] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the neural network model is quantized to obtain the first quantized neural network model according to at least one of the speed of the neural network model running on the chip to be deployed, the memory size of the chip to be deployed, the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, power consumption, temperature, and the type of instruction set (the instruction set corresponding to the assembly language supported by the chip to be deployed, the instruction set of CUDA supported by the chip to be deployed, and the instruction set corresponding to the code library supported by the chip to be deployed). In this way, it is beneficial to improve the reasoning speed of the chip to be deployed for the quantized neural network model while making full use of the hardware resources of the chip to be deployed.

[0048] It should be understood that in the embodiments of the present application, the chip to be deployed may support calculations with different bit numbers. If the chip to be deployed runs a neural network model with a high bit number at the same speed as a neural network model with a low bit number, then the neural network is quantized with a high bit number.

[0049] According to the memory size of the chip to be deployed, the CPU occupancy rate, GPU occupancy rate, memory occupancy rate, etc. of the chip to be deployed when running the neural network model, the neural network model is quantized to obtain a first quantized neural network model; this is beneficial to fully utilize the hardware resources of the chip to be deployed.

[0050] In some embodiments, if the memory size of the chip to be deployed is large, then the neural network model is quantized with a high bit number; if the memory size of the chip to be deployed is small, then the neural network model is quantized with a low bit number.

[0051] In some embodiments, if the CPU occupancy rate of the chip to be deployed is large, then the neural network model is quantized with a low bit number; if the CPU occupancy rate of the chip to be deployed is small, then the neural network model is quantized with a high bit number.

[0052] In some embodiments, if the GPU occupancy rate of the chip to be deployed is large, then the neural network model is quantized with a low bit number; if the GPU occupancy rate of the chip to be deployed is small, then the neural network model is quantized with a high bit number.

[0053] In some embodiments, if the memory occupancy rate of the chip to be deployed is large, then the neural network model is quantized with a low bit number; if the memory occupancy rate of the chip to be deployed is small, then the neural network model is quantized with a high bit number.

[0054] The neural network model is quantized according to the power consumption, temperature, etc. of the chip to be deployed when running the neural network model, to obtain a first quantized neural network model; this is beneficial to ensure that the first quantized neural network model can run efficiently under different hardware conditions.

[0055] In some embodiments, if the power consumption of the chip to be deployed is high when running the neural network model, then the neural network model is quantized with a low bit number; if the power consumption of the chip to be deployed is low, then the neural network model is quantized with a high bit number.

[0056] In some embodiments, if the temperature of the chip to be deployed is high when running the neural network model, then the neural network model is quantized with a low bit number; if the temperature of the chip to be deployed is low, then the neural network model is quantized with a high bit number.

[0057] It should be understood that in the embodiment of the present application, the speed at which the chip to be deployed runs the quantized neural network with the instruction set corresponding to the assembly language supported by the chip to be deployed may be different from the speed at which the chip to be deployed runs the quantized neural network with the instruction set of CUDA supported by the chip to be deployed, and the speed at which the chip to be deployed runs the quantized neural network with the instruction set corresponding to the code library supported by the chip to be deployed. Then, according to the nature of the instruction set when the chip to be deployed runs the neural network model, the neural network model is quantized to obtain the first quantized neural network model; this is beneficial to realize the functions corresponding to the neural network model efficiently and with low power consumption.

[0058] In some embodiments, Figure 2 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 1 ,like Figure 2 As shown, the first quantized neural network model can be obtained through the following steps 201 to 204:

[0059] Step 201, performing a second training on the neural network model; wherein the number of training times of the second training is greater than or equal to 1;

[0060] Step 202, identifying the types of layers of the neural network model; wherein the types of layers include: convolutional layer, fully connected layer;

[0061] Step 203, obtaining a Hessian matrix corresponding to the convolutional layer according to the layer characteristics of the convolutional layer obtained by the second training; wherein the layer characteristics include: eigenvalues ​​of the convolutional layer and gradients of the convolutional layer;

[0062] Step 204: According to the modulus of the Hessian matrix, the number of quantization bits of at least one of the convolutional layers is adjusted, and the number of quantization bits of at least one of the fully connected layers is adjusted to obtain the first quantized neural network model.

[0063] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the neural network model is subjected to a second training; the convolution layer and the fully connected layer of the neural network model are identified; the Hessian matrix corresponding to the convolution layer is obtained according to the eigenvalue of the convolution layer and the gradient of the convolution layer obtained by the second training; the quantization bit number of at least one convolution layer is adjusted according to the modulus of the Hessian matrix, and the quantization bit number of at least one fully connected layer is adjusted to obtain the first quantized neural network model. In this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding convolution layer to quantization, the quantization bit number of the convolution layer is adjusted according to the value of the Hessian matrix, that is, the quantization bit number of the convolution layer is adjusted according to the sensitivity of the convolution layer to quantization, and the fully connected layer is a layer that is relatively insensitive to quantization relative to the convolution layer, so the quantization bit number of the fully connected layer is uniformly adjusted; thereby achieving a balance between the accuracy and running speed of the quantized neural network model, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0064] In an embodiment of the present application, the second training is used to pre-quantize the neural network model to obtain a first quantized neural network model, and the first training is used to dynamically quantize the first quantized neural network model to obtain a dynamically quantized neural network model.

[0065] In an embodiment of the present application, the Hessian matrix is ​​calculated based on the eigenvalues ​​of the layers in the neural network model and the gradient of the layers, and the modulus of the Hessian matrix is ​​used to characterize the sensitivity of the corresponding layer to quantization. The larger the modulus of the Hessian matrix, the more sensitive the corresponding layer is to quantization, and vice versa, it is relatively insensitive to quantization. Quantization sensitivity refers to the extent to which the performance (such as precision, accuracy, etc.) of a certain layer in the neural network model decreases relative to that before quantization after quantization. If the performance of a certain layer decreases significantly after quantization, the layer is said to be sensitive to quantization; conversely, if the performance decreases slightly or remains essentially unchanged, the layer is said to be relatively insensitive to quantization.

[0066] For example, as a possible implementation, Figure 3 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 2 ,like Figure 3 As shown, the first quantized neural network model can be obtained through the following steps 301 to 307:

[0067] Step 301, performing second training.

[0068] Among them, the training object is the neural network model.

[0069] Step 302, determining the categories of different layers.

[0070] Identify the fully connected layers and convolutional layers.

[0071] Step 303: quantize the fully connected layer to 4 bits.

[0072] Step 304, calculate the Hessian matrix of each feature layer (ie, convolutional layer).

[0073] Step 305: if the value is less than the threshold, it is configured as 4-bit quantization.

[0074] That is, when the modulus of the Hessian matrix is ​​smaller than the threshold, the convolutional layer is quantized with 4 bits.

[0075] Step 306: greater than or equal to the threshold is an 8-bit quantization configuration.

[0076] That is, when the modulus of the Hessian matrix is ​​greater than or equal to the threshold, the convolutional layer is quantized with 8 bits.

[0077] Step 307, obtaining a first quantized neural network model.

[0078] In some embodiments, obtaining the first quantized neural network model further includes: quantizing the neural network model according to a preset quantization rule to obtain the first quantized neural network model.

[0079] It should be understood that in the embodiments of the present application, the preset quantization rule is not limited. In some embodiments, the preset quantization rule can be to quantize the layers of the neural network model with 4 bits, or to quantize the layers of the neural network model with 8 bits; wherein the layers of the neural network model include at least one of the following: convolutional layer, fully connected layer, activation layer, pooling layer, normalization layer, feature fusion layer, sampling layer (for example: upsampling or downsampling, etc.), etc.

[0080] In some embodiments, the categories of layers further include: an activation layer; Figure 4 A schematic diagram of an implementation process of obtaining a first quantized neural network model provided in an embodiment of the present application Figure 3 ,like Figure 4 As shown, the first quantized neural network model can be obtained through the following steps 401 to 403:

[0081] Step 401, statistically analyzing the data distribution characteristics of the activation layer obtained by the second training to obtain the size range of the characteristic value of the activation layer;

[0082] Step 402, when the size range of the characteristic value of the activation layer is greater than the range threshold, adjusting the activation function corresponding to the activation layer in the first quantized neural network model to a truncated activation function to obtain an optimized neural network model;

[0083] Step 403: Use the optimized neural network model as the first quantized neural network model.

[0084] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, the data distribution characteristics of the activation layer obtained by the second training are statistically analyzed to obtain the size range of the eigenvalues ​​of the activation layer; when the size range of the eigenvalues ​​of the activation layer is greater than the range threshold, the activation function corresponding to the activation layer in the first quantized neural network model is adjusted to a truncated activation function to obtain an optimized neural network model; the optimized neural network model is used as the first quantized neural network model. In this way, when the size range of the eigenvalues ​​of the activation layer is greater than the range threshold, the maximum and minimum values ​​of the eigenvalues ​​are limited by the truncated activation function, and the size range of the eigenvalues ​​is reduced, so that in the process of dynamic quantization, the loss of the eigenvalues ​​after dynamic quantization is less, thereby improving the stability of the neural network model after dynamic quantization; and because the size range of the eigenvalues ​​is reduced, the number of quantization bits required is also less, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0085] It should be understood that in the embodiments of the present application, the range threshold is not limited, and the range threshold value can be preset or determined according to the specific situation of the neural network model, such as: the structure of the neural network model, the number of parameters of the neural network model, the performance requirements of the neural network model, etc. The truncated activation function is also not limited. In some embodiments, the truncated activation function is a Rectified Linear Unit 6 (ReLU6).

[0086] In step 102, a first training is performed on the first quantized neural network model; the number of training times corresponding to the first training is greater than or equal to 3.

[0087] In step 103, during the first training process, the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the loss change value of at least two adjacent trainings to obtain a second quantized neural network model; the second quantized neural network model is used as the first quantized neural network model to continue the first training, and the above steps are iteratively performed until the first training is completed to obtain a dynamically quantized neural network model; the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed can realize the functions corresponding to the neural network model based on the dynamically quantized neural network model.

[0088] It should be understood that in the embodiment of the present application, the layer in "adjusting the number of quantization bits of at least one layer of the neural network model after the first quantization" can be a convolutional layer.

[0089] In some embodiments, the number of quantization bits of at least one layer of the neural network model after the first quantization can also be adjusted according to the loss change value of N consecutive trainings; wherein N is greater than or equal to 3.

[0090] It should be understood that in the embodiments of the present application, there is no limitation on the N, and the N may be preset or set according to the actual training situation of the first quantized neural network model, and in the first training process, the N may be variable or consistent.

[0091] Exemplarily, as a possible implementation, in the first one-third of the training process, N is 5; in the middle one-third of the training process, N is 10; and in the last one-third of the training process, N is 20.

[0092] It should be understood that the above embodiments are used to explain the technical solution of the present application, rather than to limit the technical solution of the present application.

[0093] In some embodiments, the adjusting the number of quantization bits of at least one layer of the first quantized neural network model also includes: during the first training process, according to the accuracy change value between the accuracy of the first quantized neural network model obtained after training and the accuracy of the neural network model, adjusting the number of quantization bits of at least one layer of the first quantized neural network model.

[0094] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, during the first training process, the quantization bit number of at least one layer of the first quantized neural network model is adjusted according to the accuracy change value of the first quantized neural network model obtained by training and the accuracy change value of the neural network model. In this way, the quantization position of each layer is dynamically adjusted according to the loss change value and the accuracy change value obtained by training, thereby further ensuring the performance of the dynamically quantized neural network model while reducing the computational complexity of the model, thereby further reducing the computing power and memory requirements of the chip to be deployed.

[0095] In some embodiments, adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to the accuracy change value of the accuracy of the first quantized neural network model obtained through training and the accuracy of the neural network model includes:

[0096] When the accuracy change value is greater than or equal to the accuracy threshold, a Hessian matrix corresponding to the layer is obtained according to the layer characteristics of the layer of the first quantized neural network model obtained by current training; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer;

[0097] When the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits;

[0098] When the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to a second number of quantization bits; and the second number of quantization bits is greater than the first number of quantization bits.

[0099] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, when the accuracy change value is greater than or equal to the accuracy threshold, the Hessian matrix corresponding to the layer is obtained according to the characteristic value of the layer of the first quantized neural network model obtained by the current training and the gradient of the layer; the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the modulus of the Hessian matrix. In this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding layer to quantization, the number of quantization bits of the layer is adjusted according to the value of the Hessian matrix, that is, the number of quantization bits of the layer is adjusted according to the sensitivity of the layer to quantization; thus, for the quantization-sensitive layer, a larger number of quantization bits is set for it; for the relatively insensitive layer, a smaller number of quantization bits is set for it; that is, through the hybrid quantization method, the quantized neural network model is balanced in terms of accuracy and running speed, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0100] It should be understood that in the embodiments of the present application, the accuracy threshold is not limited. The accuracy threshold value can be preset or determined based on the specific circumstances of the neural network model, such as: the structure of the neural network model, the number of parameters of the neural network model, the performance requirements of the neural network model, etc.

[0101] It should be understood that in the embodiments of the present application, the accuracy of the neural network model is used to measure the performance of the model on a given data set. The calculation formula for accuracy is: accuracy = number of correctly predicted samples / total number of samples; the accuracy of the neural network model is usually used to indicate the degree of consistency between the model prediction results and the actual results.

[0102] In an embodiment of the present application, the number of quantization bits may include one of 8 bits, 4 bits, 3 bits, 2 bits, etc.

[0103] In some embodiments, adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to the loss change value of at least two adjacent trainings includes:

[0104] When the loss change value is less than or equal to the loss threshold, a Hessian matrix corresponding to the layer is obtained according to the layer characteristics of the layer of the first quantized neural network model obtained in the latter training of two adjacent trainings; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer;

[0105] When the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits;

[0106] When the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to a second number of quantization bits; and the second number of quantization bits is greater than the first number of quantization bits.

[0107] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, when the loss change value is less than or equal to the loss threshold, the Hessian matrix corresponding to the layer is obtained according to the characteristic value of the layer of the first quantized neural network model obtained in the latter training of two adjacent trainings and the gradient of the layer; according to the modulus of the Hessian matrix, the quantization bit number of at least one layer of the first quantized neural network model is adjusted. In this way, since the modulus of the Hessian matrix can characterize the sensitivity of the corresponding layer to quantization, the quantization bit number of the layer is adjusted according to the value of the Hessian matrix, that is, the quantization bit number of the layer is adjusted according to the sensitivity of the layer to quantization; thus, for the quantization-sensitive layer, a larger quantization bit number is set for it; for the relatively insensitive layer, a smaller quantization bit number is set for it; that is, through the hybrid quantization method, the quantized neural network model is balanced in terms of accuracy and running speed, thereby reducing its computational complexity while ensuring the performance of the quantized neural network model, thereby reducing the computing power and memory requirements of the chip to be deployed.

[0108] It should be understood that in the embodiments of the present application, the loss threshold is not limited. The loss threshold can be preset or determined based on the specific circumstances of the neural network model, such as: the structure of the neural network model, the number of parameters of the neural network model, the performance requirements of the neural network model, etc.

[0109] In the embodiment of the present application, the modulus threshold, the first quantization bit number, and the second quantization bit number are not limited. The modulus threshold, the first quantization bit number, and the second quantization bit number can be preset, or can be determined according to the specific situation of the neural network model, for example: the structure of the neural network model, the parameter quantity of the neural network model, the performance requirements of the neural network model, etc. In some embodiments, the first quantization bit number is 4 bits, and the second quantization bit number is 8 bits.

[0110] Exemplarily, as a possible implementation, if the value of N is 5, the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the loss change value of 5 consecutive trainings. Then, when the loss change value of the loss value obtained by the Xth training and the loss value obtained by the 4+Xth training is less than or equal to the loss threshold, the Hessian matrix corresponding to the layer is obtained according to the layer characteristics of the layer of the first quantized neural network model obtained by the 4+Xth training; wherein the layer characteristics include: the eigenvalue of the layer, the gradient of the layer; when the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits; when the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the second number of quantization bits; the second number of quantization bits is greater than the first number of quantization bits; wherein X is greater than or equal to 1.

[0111] In some embodiments, the layer characteristics also include: the computational complexity of the layer, the precision of the layer; when the loss change value is less than or equal to the loss threshold, the method also includes: performing statistical analysis on the data distribution characteristics of the layer of the first quantized neural network model obtained from the latter training of two adjacent trainings to obtain the size range of the eigenvalues ​​of the layer; adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to at least one of the following: the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer.

[0112] It can be understood that in the quantization method of the neural network model provided in the embodiment of the present application, not only the first quantized neural network model is dynamically quantized based on the Hessian matrix corresponding to the layer, but also the first quantized neural network model is dynamically quantized based on the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer. In this way, since the computational complexity of the layer directly affects the computing power requirements of the quantized neural network model for the chip to be deployed, the precision of the layer and the size range of the eigenvalues ​​of the layer directly affect the performance of the quantized neural network model, the first quantized neural network model is dynamically quantized based on the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer, which is beneficial to further ensure the performance of the dynamically quantized neural network model while reducing the computational complexity of the model, thereby further reducing the computing power and memory requirements of the chip to be deployed.

[0113] It should be understood that in the embodiments of the present application, the higher the computational complexity of a layer, the higher the requirement of the layer for the computing power of the chip to be deployed; therefore, the higher the computational complexity of the layer, the lower the number of quantization bits set for the layer; the smaller the accuracy of the layer, the higher the accuracy of the neural network model after quantization; therefore, the lower the accuracy of the layer, the higher the number of quantization bits set for the layer; the larger the size range of the eigenvalues ​​of the layer, the more content is lost after quantization; therefore, the larger the size range of the eigenvalues ​​of the layer, the higher the number of quantization bits set for the layer.

[0114] Exemplarily, as a possible implementation method, the layer characteristics also include: the computational complexity of the layer, the precision of the layer; when the loss change value is less than or equal to the loss threshold, the method also includes: if the value of N is 5, statistically analyzing the data distribution characteristics of the layer of the first quantized neural network model obtained by the 4+Xth training, and obtaining the size range of the eigenvalues ​​of the layer; adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to at least one of the following: the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer.

[0115] For example, as a possible implementation, Figure 5 A schematic diagram of the implementation process of a quantization method of a neural network model provided in an embodiment of the present application Figure 2 The method includes the following steps 501 to 507:

[0116] Step 501, start the second training;

[0117] Step 502, the activation function is replaced with ReLU6;

[0118] Step 503, start the first training;

[0119] Step 504, ending several rounds of training;

[0120] Step 505, adjusting the number of quantization bits;

[0121] Step 506, determine whether the first training is finished; if so, execute step 503, otherwise execute step 507;

[0122] Step 507: The first training is completed, and a dynamically quantized neural network model is obtained.

[0123] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0124] In order to reduce resource consumption while maintaining the accuracy of the model (i.e., an example of a neural network model), low-bit quantization technology has emerged. The embodiment of the present application provides a quantization method for a neural network model; the method solves the following problems;

[0125] (1) Model accuracy loss problem

[0126] Low-bit quantization technology reduces storage requirements and computational complexity by reducing the number of bits used to represent model parameters and activation values. However, this reduction in bits will lead to a loss of model information, resulting in a decrease in accuracy. Especially in complex tasks, this loss of accuracy will significantly affect the performance of the model.

[0127] The embodiment of the present application introduces a dynamic quantization strategy and a hybrid quantization mechanism, and adopts differentiated quantization bits for different layers and parameter types, thereby minimizing the loss of precision. In addition, by using a truncated activation function and limiting the parameters within a certain range, the parameter error caused by quantization can be reduced, thereby improving the accuracy of the quantized model.

[0128] (2) Training instability problem

[0129] During the low-bit quantization process, due to the reduced representation accuracy of the parameters, the model may experience problems such as gradient oscillation and convergence difficulties during training, resulting in an unstable training process or even failure to converge to a valid solution.

[0130] The embodiment of the present application uses a truncated activation function to control the parameters before quantization within a certain range, so that the parameter distribution after each quantization is relatively stable, thereby improving the stability of quantization training. In addition, the dynamic quantization strategy ensures that the model can maintain the necessary calculation accuracy and gradient stability during the training process by adaptively adjusting the number of quantization bits.

[0131] (3) Poor hardware compatibility

[0132] The performance of low-bit quantization technology varies greatly on different hardware platforms. Some hardware platforms may not be friendly to low-bit computing, resulting in low model reasoning efficiency or increased energy consumption, making it difficult to meet actual application needs.

[0133] The embodiment of the present application specially designs hardware-friendly optimization technology, which enables the quantization model to run efficiently on a variety of hardware platforms through hardware adaptive modules and optimized instruction sets for specific platforms. Whether on a high-performance GPU, tensor processing unit (TPU), or a resource-constrained mobile device, the quantization model provided by the embodiment of the present application can make full use of hardware resources and provide optimal performance.

[0134] (4) Insufficient generalization ability

[0135] Related low-bit quantization methods have poor generalization capabilities on different tasks or data sets, that is, the trained quantization model performs poorly on unknown data and is difficult to apply to a wide range of scenarios.

[0136] The embodiment of the present application uses a hybrid quantization mechanism to adaptively adjust the quantization strategy for different levels and task requirements, thereby improving the generalization ability of the model. During the training process, the model uses a truncated activation function to maintain high accuracy while adapting to different data distributions, thereby enhancing the performance of the model in various scenarios.

[0137] The specific implementation strategies of this application are mainly centered around four core parts: dynamic quantization strategy, hybrid quantization mechanism, quantization error compensation mechanism, and hardware-friendly optimization. These strategies provide detailed steps and technical details on how to implement low-bit quantization models in practical applications.

[0138] Implementation steps for dynamic quantization strategy and hybrid quantization mechanism:

[0139] Step 11. Model level analysis:

[0140] First, the hierarchical structure of the target neural network model (i.e., an example of a neural network model) is analyzed in detail to identify the characteristics of each layer, including the size of the feature map of input and output, computational complexity, sensitivity to accuracy, etc. These characteristics will be used in the subsequent dynamic quantization decision process.

[0141] An adaptive analysis algorithm is used to perform statistical analysis on the gradients and activation distributions of each layer to identify layers that are sensitive to quantization and layers that are relatively insensitive.

[0142] Step 12. Dynamic quantization bit adjustment:

[0143] According to the results of hierarchical analysis, the number of quantization bits of each layer is dynamically adjusted. For example, for the convolutional layer, 8-bit quantization can be used, while for the insensitive fully connected layer, 4-bit or lower quantization bits can be used (hybrid quantization mechanism).

[0144] During the training (i.e., the first training), dynamic adjustments can be made not only based on pre-set rules (i.e., an example of a preset quantization rule), but also based on real-time training feedback. The system monitors the training loss and accuracy of the model in real time and adjusts the number of quantization bits based on these indicators.

[0145] Step 13. Adaptively adjust the number of parameter quantization bits during training:

[0146] In the dynamic quantization process, after every few trainings, the size of the Hessian matrix feature map, computational complexity, sensitivity to precision, etc. of each layer can be recalculated. Based on this, the number of quantization bits of each layer is dynamically adjusted to ensure that the model can effectively adapt to the new quantization strategy.

[0147] It should be understood that in the embodiments of the present application, the dynamic adjustment of the number of quantization bits not only depends on the characteristics of the layer, but can also be combined with the task complexity and the characteristics of the input data to form a flexible adaptive quantization framework.

[0148] The real-time training feedback mechanism is the key to ensuring the effectiveness of the dynamic quantization strategy. By monitoring the changes in accuracy and loss during training, the quantization strategy can be dynamically adjusted to avoid a sudden drop in model performance.

[0149] The core of hybrid quantization lies in the in-depth understanding of model hierarchy and parameter types. Through differentiated quantization strategies, it can not only retain the expressive power of the model but also effectively reduce the computational complexity.

[0150] The layered combination strategy is the key to the hybrid quantization mechanism. By rationally combining quantization methods at different levels, the efficiency and accuracy of the overall model are ensured.

[0151] The implementation steps for adjusting the activation function of the layer:

[0152] Step 21. During the quantization training process, the activation function is changed to a truncated activation function:

[0153] During the model training process, the original activation function of the model is replaced with a truncated activation function. For example, the original ReLU is replaced with ReLU6, which ensures that the maximum and minimum values ​​of the output parameters of each feature layer are within a certain range.

[0154] Among them, the ReLU activation function formula is:

[0155] x=max(0,x)

[0156] ReLU6 activation function formula:

[0157]

[0158] It should be understood that in the embodiment of the present application, by limiting the maximum and minimum values ​​output by each feature layer, the sensitivity of the quantized parameters can be improved during quantization.

[0159] Implementation steps for hardware-friendly optimization:

[0160] Step 31. Hardware platform analysis:

[0161] Before model deployment, the target hardware platform (i.e., an example of the chip to be deployed) is first analyzed, including hardware architecture, instruction set support, computing power, memory capacity, etc. Based on this information, a suitable quantization strategy is determined. For example, on a GPU platform, its powerful parallel computing capabilities can be used for efficient low-bit operations, while on mobile devices, it is necessary to consider the optimization of energy efficiency.

[0162] Step 32. Hardware Adaptive Quantization:

[0163] Design and implement a hardware adaptive module that automatically adjusts the quantization strategy based on the characteristics of the hardware platform. For example, on hardware that supports the Single Instruction, Multiple Data (SIMD) instruction set, the quantization algorithm can be adjusted to a form that is more suitable for parallel computing. In actual deployment, this module will dynamically adjust the quantization strategy based on real-time hardware performance feedback (such as power consumption, temperature, computing speed, etc.) to ensure that the model can run efficiently under different hardware conditions.

[0164] Step 33. Optimize instruction set generation:

[0165] Design and generate optimized instruction sets for different hardware platforms so that the quantized model can fully utilize hardware resources. For example, for mobile devices based on ARM architecture, generate specific low-bit operation instruction sets to improve the model reasoning speed. The generation of optimized instruction sets is based on the characteristics of the hardware platform and combined with the requirements of the quantized algorithm to ensure that the hardware can complete the task efficiently and with low power consumption when performing low-bit operations.

[0166] It should be understood that in the embodiments of the present application, the core of hardware-friendly optimization lies in understanding and utilizing the characteristics of the hardware platform, and ensuring the efficient operation of the model on different platforms through adaptive modules and optimized instruction sets. The real-time performance feedback mechanism is an important part of the hardware adaptive module, which can dynamically adjust the quantization strategy according to the actual hardware conditions to achieve optimal performance.

[0167] It can be understood that in the embodiments of the present application, a complete set of low-bit quantization technology solutions is provided through detailed implementation strategies. Through innovative technologies such as dynamic quantization, hybrid quantization, error compensation, and hardware-friendly optimization, not only the problems of precision loss and training instability in existing low-bit quantization technologies are effectively solved, but also the efficient operation of the model on different hardware platforms is ensured. These implementation strategies provide a solid technical foundation for the practical application of low-bit quantization models.

[0168] The key points of a quantization method for a neural network model provided in an embodiment of the present application focus on how to minimize the accuracy loss of the model, improve training stability, enhance hardware adaptability, and improve the generalization ability of the model during the low-bit quantization process.

[0169] These key points include the following aspects:

[0170] 1. Dynamic quantitative strategy:

[0171] During model training and inference, the number of quantization bits of each layer is dynamically adjusted, and the optimal number of quantization bits is adaptively selected according to the characteristics of each layer and task requirements to ensure the best balance between accuracy and computational efficiency between different layers. Different quantization bits or methods are used for different levels or parameter types of the model (such as convolution kernels, fully connected layer weights, activation values, etc.) to minimize storage and computing requirements while maintaining or improving model accuracy.

[0172] The dynamic quantization mechanism provided in the embodiment of the present application includes a dynamic adjustment algorithm and a decision-making process, in particular, a strategy for dynamically determining the optimal number of quantization bits based on the model hierarchy and data distribution characteristics.

[0173] 2. Adjust the activation function of each layer:

[0174] During the quantization process, the quantization range of a certain layer is updated by tracking and feeding back the quantization error in real time and adjusting the activation function update truncation.

[0175] In an embodiment of the present application, the quantization activation function is adjusted from the original ReLU function to the ReLU6 function, which limits the output range of each parameter feature layer.

[0176] 3. Hardware-friendly optimization:

[0177] Design quantization algorithms that are compatible with multiple hardware platforms (such as mobile devices, embedded systems, GPUs, and TPUs) so that the model can run efficiently on different hardware platforms. Through hardware adaptive optimization and instruction set optimization, the inference speed and energy efficiency of the model on different platforms can be further improved.

[0178] In the embodiments of the present application, hardware-friendly optimization technology, especially adaptive adjustment of quantization algorithms for different hardware architectures, optimized instruction set generation technology, and specific implementation methods of how to achieve efficient operation on different platforms through these technologies.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present application, which should be included in the scope of protection of the present application.

[0180] It should be noted that although the steps of the method in the present application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.; or, the steps in different embodiments may be combined into a new technical solution. Based on the foregoing embodiments, an embodiment of the present application provides a device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor may be an AI acceleration engine (such as NPU, etc.), a graphics processor (GPU), a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0181] Figure 6 A schematic diagram of a quantization device for a neural network model provided in an embodiment of the present application, such as Figure 6 As shown, the quantization device 60 includes: a pre-quantization module 601, a first training module 602 and a dynamic quantization module 603; wherein,

[0182] A pre-quantization module 601 is configured to quantize the neural network model according to the performance index of the chip to be deployed, so as to obtain a first quantized neural network model;

[0183] A first training module 602 is configured to perform a first training on the first quantized neural network model; wherein the number of training times corresponding to the first training is greater than or equal to 3;

[0184] The dynamic quantization module 603 is configured to adjust the quantization bits of at least one layer of the first quantized neural network model according to the loss change value of at least two adjacent trainings during the first training process to obtain a second quantized neural network model; continue the first training with the second quantized neural network model as the first quantized neural network model, and iteratively perform the above steps until the first training is completed to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed implements the function corresponding to the neural network model based on the dynamically quantized neural network model.

[0185] In some embodiments, the dynamic quantization module 603 is further configured to adjust the number of quantization bits of at least one layer of the first quantized neural network model according to the accuracy change value between the accuracy of the first quantized neural network model obtained through training and the accuracy of the neural network model during the first training process.

[0186] In some embodiments, the dynamic quantization module 603 is also configured to obtain the Hessian matrix corresponding to the layer according to the layer characteristics of the layer of the first quantized neural network model obtained by current training when the accuracy change value is greater than or equal to the accuracy threshold; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer; when the modulus of the Hessian matrix is ​​less than the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the first quantization bit number; when the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the second quantization bit number; the second quantization bit number is greater than the first quantization bit number.

[0187] In some embodiments, the dynamic quantization module 603 is configured to obtain the Hessian matrix corresponding to the layer according to the layer characteristics of the layer of the first quantized neural network model obtained in the latter training of two adjacent trainings when the loss change value is less than or equal to the loss threshold; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer; when the modulus of the Hessian matrix is ​​less than the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the first quantization bit number; when the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the quantization bit number of the layer corresponding to the Hessian matrix is ​​set to the second quantization bit number; the second quantization bit number is greater than the first quantization bit number.

[0188] In some embodiments, the layer characteristics also include: the computational complexity of the layer, the precision of the layer; the dynamic quantization module 603 is also configured to perform statistical analysis on the data distribution characteristics of the layer of the first quantized neural network model obtained from the latter training of two adjacent trainings to obtain the size range of the eigenvalues ​​of the layer; and adjust the number of quantization bits of at least one layer of the first quantized neural network model according to at least one of the following: the computational complexity of the layer, the precision of the layer, and the size range of the eigenvalues ​​of the layer.

[0189] In some embodiments, the pre-quantization module 601 is further configured to perform a second training on the neural network model; wherein the number of training times of the second training is greater than or equal to 1; identify the category of the layer of the neural network model; wherein the category of the layer includes: convolutional layer, fully connected layer; according to the layer characteristics of the convolutional layer obtained by the second training, obtain the Hessian matrix corresponding to the convolutional layer; wherein the layer characteristics include: the eigenvalue of the convolutional layer, the gradient of the convolutional layer; according to the modulus of the Hessian matrix, adjust the number of quantization bits of at least one of the convolutional layers, and adjust the number of quantization bits of at least one of the fully connected layers to obtain the neural network model after the first quantization.

[0190] In some embodiments, the category of the layer also includes: an activation layer; the pre-quantization module 601 is further configured to perform statistical analysis on the data distribution characteristics of the activation layer obtained by the second training to obtain a size range of the eigenvalues ​​of the activation layer; when the size range of the eigenvalues ​​of the activation layer is greater than a range threshold, the activation function corresponding to the activation layer in the first quantized neural network model is adjusted to a truncated activation function to obtain an optimized neural network model; and the optimized neural network model is used as the first quantized neural network model.

[0191] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0192] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0193] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0194] An embodiment of the present application provides an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the electronic device 70 includes a memory 701 and a processor 702. The memory 701 stores a computer program that can be run on the processor 702. When the processor 702 executes the program, the steps in the method provided in the above embodiment are implemented.

[0195] It should be noted that the memory 701 is configured to store instructions and applications executable by the processor 702, and can also cache data to be processed or processed by the processor 702 and various modules in the electronic device 70, which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0196] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0197] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0198] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0199] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0200] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0201] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0203] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0204] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0205] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0206] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0207] The methods disclosed in several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments. The features disclosed in several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

[0208] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0209] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A quantization method for a neural network model, characterized in that: The method comprises: quantizing the neural network model according to the performance index of the chip to be deployed, to obtain a first quantized neural network model; Performing a first training on the first quantized neural network model; wherein the number of training times corresponding to the first training is greater than or equal to 3; During the first training process, the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the loss change value of at least two adjacent trainings to obtain a second quantized neural network model; the first training is continued using the second quantized neural network model as the first quantized neural network model, and the above steps are iteratively performed until the first training is completed to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed can implement the function corresponding to the neural network model based on the dynamically quantized neural network model.

2. The quantization method of the neural network model according to claim 1, characterized in that: The adjusting the number of quantization bits of at least one layer of the first quantized neural network model also includes: During the first training process, the number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to the accuracy change value of the first quantized neural network model obtained through training and the accuracy of the neural network model.

3. The quantization method of the neural network model according to claim 2, characterized in that: The step of adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to the accuracy change value of the accuracy of the first quantized neural network model obtained through training and the accuracy of the neural network model comprises: When the accuracy change value is greater than or equal to the accuracy threshold, a Hessian matrix corresponding to the layer is obtained according to the layer characteristics of the layer of the first quantized neural network model obtained by current training; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer; When the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits; When the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to a second number of quantization bits; and the second number of quantization bits is greater than the first number of quantization bits.

4. The quantization method of the neural network model according to any one of claims 1 to 3, characterized in that: The step of adjusting the number of quantization bits of at least one layer of the first quantized neural network model according to the loss change values ​​of at least two adjacent trainings includes: When the loss change value is less than or equal to the loss threshold, a Hessian matrix corresponding to the layer is obtained according to the layer characteristics of the layer of the first quantized neural network model obtained in the latter training of two adjacent trainings; wherein the layer characteristics include: the eigenvalue of the layer and the gradient of the layer; When the modulus of the Hessian matrix is ​​less than the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to the first number of quantization bits; When the modulus of the Hessian matrix is ​​greater than or equal to the modulus threshold, the number of quantization bits of the layer corresponding to the Hessian matrix is ​​set to a second number of quantization bits; and the second number of quantization bits is greater than the first number of quantization bits.

5. The quantization method of the neural network model according to claim 4, characterized in that: The layer characteristics further include: the computational complexity of the layer and the accuracy of the layer; and when the loss change value is less than or equal to the loss threshold, the method further includes: Performing statistical analysis on the data distribution characteristics of the layer of the first quantized neural network model obtained from the latter of two adjacent trainings to obtain a size range of the characteristic value of the layer; The number of quantization bits of at least one layer of the first quantized neural network model is adjusted according to at least one of the following: the computational complexity of the layer, the accuracy of the layer, and the size range of the eigenvalues ​​of the layer.

6. The quantization method of the neural network model according to any one of claims 1 to 3, characterized in that: The step of obtaining the first quantized neural network model further includes: Performing a second training on the neural network model; wherein the number of training times of the second training is greater than or equal to 1; Identify the types of layers of the neural network model; wherein the types of layers include: convolutional layer and fully connected layer; Obtaining a Hessian matrix corresponding to the convolutional layer according to the layer characteristics of the convolutional layer obtained by the second training; wherein the layer characteristics include: an eigenvalue of the convolutional layer and a gradient of the convolutional layer; According to the modulus of the Hessian matrix, the number of quantization bits of at least one of the convolutional layers is adjusted, and the number of quantization bits of at least one of the fully connected layers is adjusted to obtain the first quantized neural network model.

7. The quantization method of the neural network model according to claim 6, characterized in that: The category of the layer also includes: an activation layer; after obtaining the first quantized neural network model, the method further includes: Performing statistical analysis on the data distribution characteristics of the activation layer obtained by the second training to obtain a size range of a characteristic value of the activation layer; When the size range of the characteristic value of the activation layer is greater than the range threshold, adjusting the activation function corresponding to the activation layer in the first quantized neural network model to a truncated activation function to obtain an optimized neural network model; The optimized neural network model is used as the first quantized neural network model.

8. A quantization device for a neural network model, characterized in that: The device comprises: A pre-quantization module is configured to quantize the neural network model according to the performance index of the chip to be deployed to obtain a first quantized neural network model; A first training module is configured to perform a first training on the first quantized neural network model; wherein the number of training times corresponding to the first training is greater than or equal to 3; The dynamic quantization module is configured to adjust the quantization bits of at least one layer of the first quantized neural network model according to the loss change value of at least two adjacent trainings during the first training process to obtain a second quantized neural network model; continue the first training with the second quantized neural network model as the first quantized neural network model, and iteratively execute the above steps until the first training is completed to obtain a dynamically quantized neural network model; wherein the dynamically quantized neural network model is used to be deployed on the chip to be deployed, so that the chip to be deployed implements the function corresponding to the neural network model based on the dynamically quantized neural network model.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images