Neural network quantification method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a neural network quantification method and system, electronic equipment and a storage medium, and belongs to the technical field of computers. According to the method, activation output distribution of a target network layer of a neural network model is acquired, the activation output distribution is analyzed to determine data distribution characteristics, and a corresponding quantification strategy is determined according to the data distribution characteristics. Performing quantization processing of a first expected quantization type on the activation output value of the target network layer according to a quantization strategy to obtain an expected quantization result, and then determining a network quantization error according to the expected quantization result; and if yes, determining the quantization strategy and the first expected quantization type as quantization parameters of the target network layer so as to quantize an activation output value of the target network layer of the neural network subsequently according to the quantization parameters. On the premise of considering the reasoning precision of the neural network model, the appropriate quantization strategy is determined according to the data distribution characteristics of network activation, and the quantization efficiency of the neural network is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a neural network quantization method, system, electronic device, and storage medium. Background Art

[0002] Deep neural networks have shown excellent performance in fields such as image processing and language processing. However, as the network depth increases, the number of model parameters also grows. To deploy large networks in resource-constrained environments such as embedded devices and edge computing devices, it is usually necessary to miniaturize the model, which is one of the key technologies in algorithm deployment. Quantization is a technology that maps the weights and activation values of a model from high-precision floating-point numbers (such as FP32) to low-precision numerical values (such as int8). This method can significantly reduce the volume of the model, improve the calculation speed, reduce the memory bandwidth and storage requirements, thus solving the problem of deploying deep learning models on resource-constrained devices.

[0003] In related technologies, the parameters and activation values of the model are usually mapped from FP32 to FP16, int16, int8, or even int4. However, this full-network quantization method inevitably brings about accuracy loss, and the lower the quantization bit width, the greater the accuracy loss. For example, in object detection and classification tasks (such as the ImageNet classification task), it may result in an accuracy loss of 3% - 10%. To reduce the accuracy loss during quantization, the current mainly adopted method is mixed-precision quantization, that is, different quantization bit widths are used for the parameters and activation values of each layer in the model. Higher bit widths are used for network layers with higher quantization sensitivity to balance model miniaturization and accuracy loss. For example, the model can be first quantized to int8 for the whole network, the output error of each layer is statistically calculated, and then the quantization type of the layers with larger errors is manually increased to FP16 to reduce the accuracy loss. However, this method requires manual judgment of the quantization sensitivity of each layer and adjustment of the quantization type of the corresponding layer, and repeated operations until a satisfactory quantized model is obtained. The above quantization process still relies on human judgment to select the network layers that need to be mixed-quantized, and the quantization efficiency is low. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a neural network quantization method, system, electronic device, and storage medium, aiming to improve the quantization efficiency of the neural network on the premise of considering the inference accuracy of the neural network model.

[0005] To achieve the above purpose, on the one hand, an embodiment of this application proposes a neural network quantization method, including the following steps:

[0006] Obtain the activation output distribution of the target network layer of the neural network model;

[0007] Analyze the activation output distribution to determine the data distribution characteristics, and determine the corresponding quantization strategy according to the data distribution characteristics, where the quantization strategy includes a quantization method;

[0008] Perform quantization processing of the first expected quantization type on the activation output values of the target network layer according to the quantization strategy to obtain an expected quantization result;

[0009] Determine the network quantization error according to the expected quantization result;

[0010] Judge whether the network quantization error is less than the error expectation threshold. When the network quantization error is less than the error expectation threshold, determine the quantization strategy and the first expected quantization type as the quantization parameters of the target network layer, so as to quantize the activation output values of the target network layer of the neural network according to the quantization parameters.

[0011] In some embodiments, the neural network quantization method further includes the following steps:

[0012] Obtain the weight data distribution of the neural network model according to the weight values of each network layer of the neural network model;

[0013] Perform quantization processing of the second expected quantization type on each weight value of the neural network model by a linear mapping method according to the maximum value and the minimum value in the weight data distribution.

[0014] In some embodiments, the first expected quantization type is obtained through the following steps:

[0015] Obtain a quantization type sequence, where the quantization type sequence includes preset quantization types arranged in ascending order of data precision;

[0016] Quantize the target network layer using the first preset quantization type in the quantization type sequence;

[0017] Determine the quantization sensitivity according to the quantized target network layer, and judge whether the quantization sensitivity is greater than the sensitivity expectation threshold;

[0018] When the quantization sensitivity is greater than the sensitivity expectation threshold, quantize the target network layer using the next preset quantization type in the quantization type sequence, and repeat the steps of determining the quantization sensitivity according to the quantized target network layer and judging whether the quantization sensitivity is greater than the sensitivity expectation threshold;

[0019] When the quantization sensitivity is less than or equal to the sensitivity expectation threshold, use the current preset quantization type as the first expected quantization type.

[0020] In some embodiments, obtaining the activation output distribution of the target network layer of the neural network model includes the following steps:

[0021] Obtain a calibration data set, where the calibration data set includes a plurality of unlabeled data;

[0022] Input the calibration data set into the neural network model for inference and extract the output of the target network layer in the neural network model to obtain the activation output distribution of the target network layer.

[0023] In some embodiments, the data distribution characteristics include variance, skewness, and kurtosis. Analyzing the activation output distribution to determine the data distribution characteristics includes the following steps:

[0024] Calculate the average of the squares of the differences between each data in the activation output distribution and the average to obtain the variance;

[0025] Calculate the square root of the variance to obtain the standard deviation;

[0026] Calculate the third central moment of the activation output distribution and determine the skewness according to the third central moment and the standard deviation;

[0027] Calculate the fourth central moment of the activation output distribution and determine the kurtosis according to the fourth central moment and the standard deviation.

[0028] In some embodiments, determining the corresponding quantization strategy according to the data distribution characteristics includes the following steps:

[0029] Judge whether the data distribution characteristics meet the first preset condition, where the first preset condition is that the skewness is less than the first preset value, the variance is less than the second preset value, and the kurtosis is less than the third preset value;

[0030] When the data distribution characteristics meet the first preset condition, determine that the quantization method in the quantization strategy is the minimum-maximum unsaturated quantization method;

[0031] When the data distribution characteristics do not meet the first preset condition, judge whether the data distribution characteristics meet the second preset condition, where the second preset condition is that the skewness is greater than the first preset value or the kurtosis is greater than the third preset value;

[0032] When the data distribution characteristics meet the second preset condition, determine that the quantization method in the quantization strategy is the KL divergence saturated quantization method;

[0033] When the data distribution characteristics do not meet the second preset condition, judge whether the data distribution characteristics meet the third preset condition, where the third preset condition is that the variance is greater than or equal to the second preset value;

[0034] When the data distribution feature satisfies the third preset condition, determine that the quantization method in the quantization strategy is the alternating direction multiplier method saturation quantization method;

[0035] When the data distribution feature does not satisfy the third preset condition, determine that the quantization method in the quantization strategy is the simple quantization method.

[0036] In some embodiments, the neural network quantization method includes the following steps:

[0037] When the result similarity is less than the similarity expectation threshold, determine that the quantization method of directly truncating single-precision floating-point numbers to half-precision floating-point numbers is the quantization parameter of the target network layer.

[0038] To achieve the above object, another aspect of the embodiments of the present application proposes a neural network quantization system, including:

[0039] A first module, configured to obtain the activation output distribution of the target network layer of the neural network model;

[0040] A second module, configured to analyze the activation output distribution to determine the data distribution feature, and determine the corresponding quantization strategy according to the data distribution feature, where the quantization strategy includes a quantization method;

[0041] A third module, configured to perform quantization processing of the first expected quantization type on the activation output value of the target network layer according to the quantization strategy to obtain an expected quantization result;

[0042] A fourth module, configured to determine the network quantization error according to the expected quantization result;

[0043] A fifth module, configured to determine whether the network quantization error is less than the error expectation threshold. When the network quantization error is less than the error expectation threshold, determine the quantization strategy and the first expected quantization type as the quantization parameters of the target network layer, so as to quantize the activation output value of the target network layer of the neural network according to the quantization parameters.

[0044] To achieve the above object, another aspect of the embodiments of the present application proposes an electronic device, where the electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory. When the program is executed by the processor, the method described in the above embodiments is implemented.

[0045] To achieve the above object, another aspect of the embodiments of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiments.

[0046] The neural network quantization method, system, electronic device and storage medium proposed in the present application obtain the activation output distribution of the target network layer of the neural network model, analyze the activation output distribution to determine the data distribution characteristics, and determine the corresponding quantization strategy according to the data distribution characteristics. The activation output values of the target network layer are quantized by the first expected quantization type according to the quantization strategy to obtain the expected quantization result, and then the network quantization error is determined according to the expected quantization result. When the network quantization error is less than the error expectation threshold, the quantization strategy and the first expected quantization type are determined as the quantization parameters of the target network layer, so as to subsequently quantize the activation output values of the target network layer of the neural network according to the quantization parameters. On the premise of considering the inference accuracy of the neural network model, the present application determines a suitable quantization strategy according to the data distribution characteristics of the network activation, and improves the quantization efficiency of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of the neural network quantization method provided by the embodiments of the present application;

[0048] Figure 2 is a flowchart of the neural network quantization method provided by another embodiment of the present application;

[0049] Figure 3 is Figure 1 a flowchart of the method for obtaining the first expected quantization type in step S103 in;

[0050] Figure 4 is Figure 1 a flowchart of step S101 in;

[0051] Figure 5 is Figure 1 a flowchart of the step of analyzing the activation output distribution to determine the data distribution characteristics in step S102 in;

[0052] Figure 6 is Figure 1 a flowchart of the step of determining the corresponding quantization strategy according to the data distribution characteristics in step S102 in;

[0053] Figure 7 is Figure 1 a flowchart of step S104 in;

[0054] Figure 8It is a schematic structural diagram of the neural network quantization system provided by an embodiment of the present application;

[0055] Figure 9 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application.

[0056] Figure 10 It is a schematic diagram of the quantization parameter selection process for the activation output of the neural network model network layer provided by an embodiment of the present application. Detailed implementation manners

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that although functional module division is performed in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order from the module division in the system or the order in the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] First, several terms involved in the present application are analyzed:

[0061] MinMax quantization is a simple and direct quantization strategy. The MinMax quantization method finds the maximum value (Max) and the minimum value (Min) in the original floating-point dataset, and then defines a linear transformation based on these two values to linearly map the original floating-point data to the range of the quantized fixed-point data. Specifically, the range (Min, Max) of the floating-point data is mapped to the range of the quantized fixed-point number, especially commonly used in the quantization process of mapping 32-bit floating-point numbers (FP32) to 8-bit integers (INT8).

[0062] KL Divergence (KLD) Quantization. KLD is an asymmetric measure used in probability theory and statistics to measure the difference between two probability distributions. It was proposed by Solomon Kullback and Richard Leibler in the 1950s. Although KLD does not satisfy the symmetry and triangle inequality of the distance definition, due to its wide application in fields such as information theory and machine learning, KLD is also known as the KL distance. The principle of the KLD quantization method is to select appropriate quantization parameters by comparing the similarity between the original data distribution and the quantized data distribution. Specifically, the KLD quantization method abstracts the numerical distribution of the original floating-point numbers and the integer numerical distribution after quantization into two probability distributions, and uses KLD to measure the similarity of these two distributions. The smaller the KLD, the more similar these two distributions are, indicating that the selected quantization parameters are more appropriate. In the process of KLD quantization, first, the distribution range of the original data needs to be determined and discretized into N intervals. Then, calculate the average value of the data in each interval as the representative value of the quantized data. Next, calculate the KLD between the original data distribution and the quantized data distribution, and select the best quantization parameters by minimizing this KLD.

[0063] ADMM (Alternating Direction Method of Multipliers) is an important method for solving separable convex optimization problems. ADMM combines the decomposability of the dual ascent method and the convergence of the augmented Lagrangian multiplier method. Through the decomposition and coordination process, it decomposes a large global problem into multiple smaller and easier-to-solve local sub-problems, and obtains the solution of the large global problem by coordinating the solutions of the sub-problems. The principle of the ADMM quantization method is to define multiple quantization objectives, including the quantization bit number, the quantization error tolerance, and the acceptable range of the model performance loss; based on the quantization objectives, construct the quantization problem of the neural network as a multi-objective optimization problem; after constructing the quantization optimization problem, the ADMM algorithm can be applied to solve this problem. The ADMM algorithm decomposes a large global quantization problem into multiple smaller and easier-to-solve local sub-problems through the decomposition and coordination process, and obtains the solution of the global problem by coordinating the solutions of the sub-problems.

[0064] EQ (EasyQuant) is a quantization method for neural network models. The EQ method adopts the strategy of error accumulation, transforming the quantization decision of the entire network into a decision-making process of a single network. By alternately optimizing the weight scaling coefficient and the activation value scaling coefficient, and using the cosine similarity as the target, the quantized model and the original floating-point model maintain a high similarity in output.

[0065] Embodiments of the present application provide a neural network quantization method, system, electronic device, and storage medium, aiming to improve the quantization efficiency of the neural network on the premise of considering the inference accuracy of the neural network model.

[0066] The neural network quantization method, system, electronic device, and storage medium provided by the embodiments of the present application will be specifically described through the following embodiments. First, the neural network quantization method in the embodiments of the present application will be described.

[0067] The neural network quantization method provided by the embodiments of the present application relates to the field of computer technology. The neural network quantization method provided by the embodiments of the present application can be applied to a terminal, or to a server side, or can also be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the neural network quantization method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] The quantization of a neural network model is divided into two scenarios: PTQ (Post Training Quantization) and QAT (Quantization Aware Training). QAT quantization quantizes a trained model and retrains it. PTQ quantization quantizes based on an existing trained model without any training on the original model. The neural network quantization method in the embodiments of this application is mainly implemented based on the PTQ scenario. Generally, the quantization objects of a neural network model include fixed weight values and continuously changing activation output values of each layer. Since the data distribution of the activation output values in the model is dynamic, a single quantization method cannot guarantee model accuracy. Therefore, the neural network quantization method in the embodiments of this application mainly automatically selects the quantization method for the activation output values, and improves the selection efficiency of the activation quantization method for network layers considering model accuracy.

[0070] Figure 1 It is an optional flowchart of the neural network quantization method provided by the embodiments of this application. Figure 1 The method in it may include but is not limited to steps S101 to S105.

[0071] Step S101, obtain the activation output distribution of the target network layer of the neural network model;

[0072] Step S102, analyze the activation output distribution to determine the data distribution characteristics, and determine the corresponding quantization strategy according to the data distribution characteristics, where the quantization strategy includes the quantization method;

[0073] Step S103, perform quantization processing of the first expected quantization type on the activation output values of the target network layer according to the quantization strategy to obtain the expected quantization result;

[0074] Step S104, determine the network quantization error according to the expected quantization result;

[0075] Step S105, determine whether the network quantization error is less than the error expectation threshold. When the network quantization error is less than the error expectation threshold, determine the quantization strategy and the first expected quantization type as the quantization parameters of the target network layer, so as to quantize the activation output values of the target network layer of the neural network according to the quantization parameters.

[0076] Steps S101 to S105 shown in the embodiments of the present application obtain the activation output distribution of the target network layer of the neural network model, analyze the activation output distribution to determine the data distribution characteristics, determine the corresponding quantization strategy according to the data distribution characteristics, perform quantization processing of the first expected quantization type on the activation output values of the target network layer according to the quantization strategy to obtain the expected quantization result, and then determine the network quantization error according to the expected quantization result. When the network quantization error is less than the error expectation threshold, the quantization strategy and the first expected quantization type are determined as the quantization parameters of the target network layer, so as to subsequently quantize the activation output values of the target network layer of the neural network according to the quantization parameters. The embodiments of the present application determine a suitable quantization strategy according to the data distribution characteristics of the network activation on the premise of considering the inference accuracy of the neural network model, and improve the quantization efficiency of the neural network.

[0077] In step S101 of some embodiments, the neural network model consists of multiple network layers. Each network layer calculates the input of the network layer according to its neuron parameters (i.e., weights and biases) to obtain the activation output value. The activation value distribution states of different network layers with different network structures in the model are different. Therefore, different quantization strategies should be adopted for different activation value distributions to reduce information loss, because the process from FP32 to int8 is also a process of information re-encoding. For example, the activation value distributions generated by different pictures are mostly similar, but when the activation values are large, their distributions become messy. At this time, appropriate quantization strategies should be adopted to quantize the activation output values of each network layer respectively. Based on this, the embodiments of the present application need to determine the quantization parameters (including quantization strategies) of each network layer so as to dynamically quantize the activation output of the corresponding network layer by using the quantization parameters during the model application process. In this embodiment, one of the network layers of the model is called the target network layer. Multiple data output by the target network layer according to multiple input data form the activation output distribution of the target network layer.

[0078] In step S102 of some embodiments, the activation output distribution is analyzed to determine the data distribution characteristics. The data distribution characteristics can characterize the data dispersion degree and outlier distribution of the network layer activation output. The corresponding quantization strategy is matched according to the data distribution characteristics. For example, for the activation output with a uniform distribution, the MinMax quantization method can be matched, which can improve the inference efficiency while ensuring the inference accuracy of the model to a certain extent. The quantization strategy of this embodiment includes a quantization method, and the quantization method can be one of the MinMax, KLD, ADMM or EQ quantization methods to be applied to the corresponding activation output distribution. The quantization strategy also includes the verification parameters required by the corresponding quantization method. The verification parameters are obtained by analyzing the activation output distribution. For example, the verification parameters can include the maximum value, minimum value and truncation threshold in the activation output distribution.

[0079] In step S103 of some embodiments, the first expected quantization type refers to the required data precision, that is, the target bit width of data quantization. For example, if the original bit width of the quantization object of the model is FP32 (Full Precise Float 32, single-precision floating point number), the first expected quantization type can be FP16 (float, half-precision floating point number), int16 (16-bit integer), int8 (8-bit integer), or int4 (4-bit integer). The first expected quantization type of the network layer can be preset in the system according to requirements. It can be understood that the first expected quantization types of different network layers of the model can be the same or different. After determining the quantization strategy for the target network layer, the activation output value of the target network layer is quantized according to the first expected quantization type using the quantization strategy, that is, the activation output value is quantized into the format of the first expected quantization type, so as to obtain the expected quantization result.

[0080] In step S104 of some embodiments, the network quantization error is calculated by comparing the expected quantization result with the unquantized activation output value. The network quantization error can measure the degree of influence of the selected quantization strategy on the network accuracy, or the quantization sensitivity of the network layer.

[0081] In step S105 of some embodiments, it is determined whether the network quantization error is less than the error expectation threshold. When the network quantization error is less than the error expectation threshold, it indicates that the influence of the selected quantization strategy on the network accuracy is small. At this time, the quantization strategy and the first expected quantization type are determined as the quantization parameters of the target network layer, so that in the subsequent model application process, the activation output value of the target network layer of the neural network is quantized using the quantization parameters of the target network layer. The quantization parameters used can balance the model accuracy and the model processing speed.

[0082] Please refer to Figure 2 , in some embodiments, the neural network quantization method of the embodiments of the present application may further include but is not limited to steps S201 to S202:

[0083] Step S201, obtaining the weight data distribution of the neural network model according to the weight values of each network layer of the neural network model;

[0084] Step S202, performing quantization processing on each weight value of the neural network model according to the maximum value and the minimum value in the weight data distribution in a linear mapping manner using the second expected quantization type.

[0085] In this embodiment, the weight parameters of the neural network model are statically distributed, and the MinMax quantization method can be directly used to perform linear mapping on them, so as to realize the quantization of each weight value. Specifically, the weight values of each network layer of the neural network model are obtained to obtain the weight data distribution of the neural network model. The minimum value and the maximum value are found from the weight data distribution, and according to the preset second expected quantization type, the weight value, the minimum value, and the maximum value are substituted into the corresponding quantization formula for calculation to obtain the quantized weight value. Exemplarily, the calculation formula is as follows:

[0086]

[0087] Among them, R represents the original floating-point value of the weight (such as FP32), Q represents the quantized fixed-point value of the weight (such as INT8), Min and Max are respectively the minimum value and the maximum value in the original weight value, n represents the size of the representation range of the quantized fixed-point number (for INT8, n = 256, but due to being signed, the actual representable range is [-127, 127]), Z represents the zero-point offset, and round(.) represents the rounding function, which is used to map the calculation result to the nearest integer value. In this embodiment, the whole-network MinMax quantization strategy is adopted for the weight values of the model, which has less precision loss for network inference, smaller quantization operation overhead, and higher quantization process efficiency.

[0088] Please refer to Figure 3 , in some embodiments, the first expected quantization type in step S103 can be obtained through but not limited to steps S301 to S305:

[0089] Step S301, obtain a quantization type sequence, where the quantization type sequence includes preset quantization types arranged in ascending order of data precision;

[0090] Step S302, quantize the target network layer using the first preset quantization type in the quantization type sequence;

[0091] Step S303, determine the quantization sensitivity according to the quantized target network layer, and judge whether the quantization sensitivity is greater than the sensitivity expected threshold;

[0092] Step S304, when the quantization sensitivity is greater than the sensitivity expected threshold, quantize the target network layer using the next preset quantization type in the quantization type sequence, and repeat step S303;

[0093] Step S305, when the quantization sensitivity is less than or equal to the sensitivity expected threshold, use the current preset quantization type as the first expected quantization type.

[0094] In this embodiment, during the whole-network quantization process, if each layer of the model is quantized using the first expected quantization type with a low bit width (such as int8), the accuracy of the model will be significantly reduced. Therefore, it is necessary to find the network layers that need to improve the quantization accuracy according to the quantization sensitivity of the network layers and perform high-bit-width (such as FP16) quantization on them to control the overall accuracy loss of the model. Specifically, a quantization type sequence is obtained, and the quantization type sequence includes preset quantization types arranged in ascending order of data precision, such as [int4, int8, int16, FP16, FP32]. Then, the activation output layer of the target network layer is quantized using the first preset quantization type int4 in the quantization type sequence, and then the quantization sensitivity is determined according to the quantized target network layer, and it is judged whether the quantization sensitivity is greater than the sensitivity expected threshold. If the quantization sensitivity is greater than the sensitivity expected threshold, the next preset quantization type int8 in the quantization type sequence is used to quantize the target network layer, and so on, until the preset quantization type corresponding to the quantization sensitivity less than or equal to the sensitivity expected threshold is matched, and this preset quantization type is used as the first expected quantization type.

[0095] It can be understood that in the above analysis process of the first expected quantization type, quantizing the activation output layer of the target network layer can refer to the target network layer after weight value quantization or the target network layer without weight value quantization, and the embodiments of the present application do not make specific limitations.

[0096] It can be understood that the second expected quantization type required for the above weight value quantization can be directly preset in the system according to requirements, or the second expected quantization type can be calculated in the same step manner as steps S301 to S305 and then preset in the system.

[0097] Please refer to Figure 4 , in some embodiments, step S101 may include but is not limited to steps S401 to S402:

[0098] Step S401, obtain a calibration data set, where the calibration data set includes a plurality of unlabeled data;

[0099] Step S402, input the calibration data set into the neural network model for inference and extract the output of the target network layer in the neural network model to obtain the activation output distribution of the target network layer.

[0100] In this embodiment, the calibration data set includes a small amount of model input data. For example, it can be image data for inputting into an image processing model, or text data for inputting into a text processing model, etc. The calibration data set can be a part of the validation set during model training. After preprocessing the calibration data set, the sample data in the calibration data set are sequentially input into the neural network model, and the activation output distributions of each network layer in the neural network model for each sample data are extracted.

[0101] Please refer to Figure 5 , in some embodiments, the data distribution characteristics include variance, skewness, and kurtosis. The step of analyzing the activation output distribution to determine the data distribution characteristics in step S102 may include, but is not limited to, steps S501 to S504:

[0102] Step S501, calculate the average of the squares of the differences between each data in the activation output distribution and the average value to obtain the variance;

[0103] Step S502, calculate the square root of the variance to obtain the standard deviation;

[0104] Step S503, calculate the third central moment of the activation output distribution, and determine the skewness based on the third central moment and the standard deviation;

[0105] Step S504, calculate the fourth central moment of the activation output distribution, and determine the kurtosis based on the fourth central moment and the standard deviation.

[0106] In this embodiment, by analyzing the activation output distribution, characteristics such as the uniformity, outlier distribution, and dispersion of the network layer output data can be determined. The above characteristics can be characterized by the variance, skewness, and kurtosis of the activation output distribution. Among them, the calculation methods of variance, skewness, and kurtosis are as follows:

[0107] The variance is expressed as:

[0108]

[0109] Among them, the i-th data in the activation output distribution, is the average value of the data in the activation output distribution, and n is the amount of data in the activation output distribution. The larger the variance, the greater the fluctuation of the data; the smaller the variance, the smaller the fluctuation of the data.

[0110] The skewness is expressed as:

[0111]

[0112] Among them, S k is the skewness, μ 3The third central moment is denoted as such, and σ is the standard deviation. The standard deviation is obtained by calculating the square root of the variance. The third central moment is obtained by calculating the cube of the difference between each data point and the mean in the activation output distribution, summing up these cubes of differences, and then dividing by the sample size (i.e., the amount of data). When the activation output distribution is right-skewed, S k > 0, and the larger the value of S k , the higher the degree of right-skewness; when the activation output distribution is left-skewed, S k < 0, and the smaller the value of S k , the higher the degree of left-skewness.

[0113] The kurtosis is expressed as:

[0114]

[0115] where, b k is the kurtosis, μ 4 is the fourth central moment, and σ is the standard deviation. The fourth central moment is obtained by calculating the fourth power of the difference between each data point and the mean in the activation output distribution, summing up these fourth powers of differences, and then dividing by the sample size (i.e., the amount of data). The kurtosis can describe the steepness of the activation output distribution. If b k < 3, the distribution is said to have insufficient kurtosis. If b k > 3, the distribution is said to have excessive kurtosis.

[0116] Please refer to Figure 6 , in some embodiments, the step of determining the corresponding quantization strategy according to the data distribution characteristics in step S102 may include, but is not limited to, steps S601 to S607:

[0117] Step S601, determining whether the data distribution characteristics satisfy a first preset condition, where the first preset condition is that the skewness is less than a first preset value, the variance is less than a second preset value, and the kurtosis is less than a third preset value;

[0118] Step S602, when the data distribution characteristics satisfy the first preset condition, determining that the quantization method in the quantization strategy is the minimum-maximum unsaturated quantization method;

[0119] Step S603, when the data distribution characteristics do not satisfy the first preset condition, determining whether the data distribution characteristics satisfy a second preset condition, where the second preset condition is that the skewness is greater than the first preset value or the kurtosis is greater than the third preset value;

[0120] Step S604, when the data distribution characteristics satisfy the second preset condition, determining that the quantization method in the quantization strategy is the KL divergence saturated quantization method;

[0121] Step S605, when the data distribution characteristics do not meet the second preset condition, determine whether the data distribution characteristics meet the third preset condition, where the third preset condition is that the variance is greater than or equal to the second preset value;

[0122] Step S606, when the data distribution characteristics meet the third preset condition, determine that the quantization method in the quantization strategy is the alternating direction method of multipliers saturation quantization method;

[0123] Step S607, when the data distribution characteristics do not meet the third preset condition, determine that the quantization method in the quantization strategy is the simple quantization method.

[0124] In this embodiment, according to the data distribution characteristics such as the variance, kurtosis, and skewness of the activation output distribution, by comparing the variance, kurtosis, and skewness with the corresponding preset values, the quantization strategy adopted by the target network layer can be judged, as follows:

[0125] S11, if abs(skewness) < 0 and variance < 0.1 and abs(kurtosis) < 3 for the activation output distribution of the target network layer, it is considered that the output data distribution is relatively uniform, and the MinMax unsaturated quantization method can be selected and the Min value and Max value are recorded to obtain the quantization strategy. If the conditions are not met, proceed to the next step.

[0126] S12, if abs(skewness) > 0 or abs(kurtosis) > 3, it is considered that the current data distribution is non-uniform, and the KLD saturation quantization method can be selected, and the truncation threshold T is recorded at the same time to obtain the quantization strategy. If the conditions are not met, proceed to the next step.

[0127] S13, if the variance > 0.1, it is considered that the data is relatively scattered and there are many outliers, and the ADMM saturation quantization method can be selected, and the truncation threshold T is recorded at the same time to obtain the quantization strategy. If the conditions are not met, proceed to the next step.

[0128] S14, if none of the above conditions are met, select the EQ quantization method, and record the activation value scaling factor S at the same time to obtain the quantization strategy.

[0129] It should be noted that MinMax quantization performs linear quantization using the minimum and maximum values of the activation values, which is suitable for cases where the data distribution is relatively uniform but is sensitive to outliers. KLD quantization measures the similarity between two distributions by comparing the KL divergence between the quantized distribution and the original distribution, thereby determining the quantization threshold, and is suitable for cases where the data distribution is non-uniform. ADMM quantization is used to solve the optimal solution problem with constraints and achieves better quantization effects by introducing penalty terms. The quantization threshold obtained by this method is usually higher than that of the KLD method because ADMM often considers more comprehensive data, so a higher-bit threshold truncation is adopted, which is suitable for scenarios that require fine control. EQ quantization regards the acquisition of the scaling factor S as a mathematical optimization problem and has a wide range of applicable scenarios.

[0130] Please refer to Figure 7 , in some embodiments, step S104 may include but is not limited to steps S701 to S702:

[0131] Step S701, perform inverse quantization of the original quantization type on the expected quantization result to obtain an inverse quantization result;

[0132] Step S702, determine the network quantization error according to the inverse quantization result and the original output result of the target network layer.

[0133] In this embodiment, taking the neural network model using FP32 data and the first expected quantization type being int8 as an example, after determining the quantization strategy for each target network layer, perform int8 quantization on each target network layer and then inverse quantize it to the FP32 output. Calculate the cosine distance between the inverse quantized FP32 output and the original FP32 model output according to the cosine distance formula, and this cosine distance can represent the network quantization error of the target network layer. Further, compare the size of the network quantization error with the preset error expectation threshold. If the network quantization error is less than the preset error expectation threshold, record the quantization parameters of this network layer (including quantization type int8, quantization method, and min, max, T(S), etc.). If the network quantization error is greater than or equal to the preset error expectation threshold, determine that the quantization method of directly truncating single-precision floating-point numbers to half-precision floating-point numbers is the quantization parameter of the target network layer.

[0134] This embodiment automatically selects a reasonable quantization type and quantization strategy for each network layer. Compared with the method of manually setting quantization parameters for each layer and repeatedly trying to obtain a suitable mixed-precision quantization effect, this embodiment actively analyzes the activation output distribution of the network, automatically selects a suitable quantization strategy according to the data distribution characteristics, and then determines the quantization parameters, so that the model can quantize the output of the network layer according to the recorded quantization parameters. Further, after determining the quantization parameters of the network layer, manual modification and fine-tuning can also be performed according to the recorded quantization parameters.

[0135] According to some embodiments of the present application, please refer to Figure 10 , and the specific embodiments of the quantization parameter selection process for the activation output of the network layer of the FP32 neural network model are as follows:

[0136] S21, provide a small amount of model verification data as a calibration data set;

[0137] S22, perform preprocessing operations on the sample data in the calibration data set. The preprocessing operations include removing outliers therein, converting the sample data into a format that conforms to the model input, etc.;

[0138] S23, input the sample data into the neural network model, and obtain the activation output values of each layer through the inference of the neural network model;

[0139] S24, analyze the data characteristics (variance, kurtosis, and skewness) of the output of each layer of the model;

[0140] S25, select a suitable quantization strategy (such as MinMax, KLD, ADMM, EQ) according to the data characteristics of the activation output of each layer to improve the performance and computational efficiency of the model. The specific quantization strategy selection process has been described in the above steps S11 to S14 and will not be elaborated here.

[0141] S26, perform int8 quantization on the output of the network through the selected quantization strategy, and quantization error is introduced during the quantization process;

[0142] S27, dequantize the quantized output into the FP32 activation output, and calculate the cosine similarity between it and the true FP32 activation output. The cosine similarity can be used as the quantization sensitivity of this network layer;

[0143] S28, compare the cosine similarity with a set threshold. If the cosine similarity is greater than or equal to the threshold, record the quantization parameters (quantization type, quantization strategy, min, max, T(S)) of the corresponding network layer, otherwise directly record the fp16 quantization type.

[0144] According to some embodiments of the present application, the embodiments of the present application achieve the adaptivity of mixed-precision quantization from two dimensions: the selection of quantization bit widths for each layer and the selection of quantization strategies. There is no need for manual judgment of the network layers that need to improve the quantization accuracy, and fully automatic mixed-precision quantization is realized, making the quantization process more reasonable and convenient. By statistically analyzing the activation output characteristics of each network layer in the model, that is, analyzing the distribution uniformity of the data, the existence and dispersion of outliers, appropriate quantization types and quantization strategy parameters are determined. Compared with the method of determining mixed quantization parameters by relying on experience and repeated attempts, the method of the embodiments of the present application can effectively improve the model quantization efficiency and reduce the accuracy loss during the quantization process. After performing int8 quantization in this embodiment, the cosine similarity between the dequantized output and the original output is calculated. If the cosine similarity is greater than a given threshold, the selected quantization type and quantization strategy can be used as quantization parameters to implement subsequent quantization; if the cosine similarity is less than the given threshold, FP16 quantization is directly used. The FP16 quantization method only needs to truncate the numerical accuracy and does not require cumbersome calculation of other parameters. This method allows more layers to be set to floating-point operations manually according to the set threshold, thus achieving a balance between accuracy and computational efficiency.

[0145] Please refer to Figure 8 , the embodiments of the present application also provide a neural network quantization system, including:

[0146] The first module is used to obtain the activation output distribution of the target network layer of the neural network model;

[0147] The second module is used to analyze the activation output distribution to determine the data distribution characteristics, and determine the corresponding quantization strategy according to the data distribution characteristics, where the quantization strategy includes quantization methods;

[0148] The third module is used to perform quantization processing of the first expected quantization type on the activation output value of the target network layer according to the quantization strategy to obtain an expected quantization result;

[0149] The fourth module is used to determine the network quantization error according to the expected quantization result;

[0150] The fifth module is used to determine whether the network quantization error is less than the error expectation threshold. When the network quantization error is less than the error expectation threshold, the quantization strategy and the first expected quantization type are determined as the quantization parameters of the target network layer to quantize the activation output value of the target network layer of the neural network according to the quantization parameters.

[0151] It can be understood that the content in the above embodiments of the neural network quantization method is applicable to the embodiments of this system. The functions specifically implemented by the embodiments of this system are the same as those of the above embodiments of the neural network quantization method, and the beneficial effects achieved are also the same as those of the above embodiments of the neural network quantization method.

[0152] An embodiment of the present application further provides an electronic device, which includes: a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory. When the program is executed by the processor, the above-mentioned neural network quantization method is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0153] Please refer to Figure 9 , Figure 9 which schematically shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0154] A processor 901, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0155] A memory 902, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the neural network quantization method of the embodiments of the present application;

[0156] An input / output interface 903, which is used to implement information input and output;

[0157] A communication interface 904, which is used to implement communication interaction between this device and other devices, and can implement communication through a wired method (such as USB, network cable, etc.) or through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

[0158] A bus 905, which transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0159] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are communicatively connected to each other inside the device through the bus 905.

[0160] The embodiments of the present application also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above neural network quantization method.

[0161] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0163] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine some steps, or different steps.

[0164] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0166] In the description of this application and the above-mentioned accompanying drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of systems or units can be in an electrical, mechanical, or other form.

[0169] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0172] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of the present application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A neural network quantization method, characterized in that: The following steps are involved: Get the activation output distribution of the target network layer of the neural network model; Analyzing the activation output distribution to determine data distribution characteristics, and determining a corresponding quantization strategy according to the data distribution characteristics, wherein the quantization strategy includes a quantization method; Performing a quantization process of a first expected quantization type on the activation output value of the target network layer according to the quantization strategy to obtain an expected quantization result; Determine a network quantization error according to the expected quantization result; Determine whether the network quantization error is less than an expected error threshold. When the network quantization error is less than the expected error threshold, determine the quantization strategy and the first expected quantization type as quantization parameters of the target network layer to quantize the activation output value of the target network layer of the neural network according to the quantization parameters.

2. The neural network quantization method according to claim 1, characterized in that: The neural network quantization method further includes the following steps: Obtaining weight data distribution of the neural network model according to weight values ​​of each network layer of the neural network model; According to the maximum value and the minimum value in the weight data distribution, each weight value of the neural network model is quantized into a second desired quantization type by linear mapping.

3. The neural network quantization method according to claim 1, characterized in that: The first desired quantization type is obtained by the following steps: Acquire a quantization type sequence, wherein the quantization type sequence includes preset quantization types arranged from low to high according to data precision; quantizing the target network layer using a first preset quantization type in the quantization type sequence; Determine a quantization sensitivity according to the quantized target network layer, and judge whether the quantization sensitivity is greater than an expected sensitivity threshold; When the quantization sensitivity is greater than the expected sensitivity threshold, the target network layer is quantized using the next preset quantization type in the quantization type sequence, and the steps of determining the quantization sensitivity according to the quantized target network layer and judging whether the quantization sensitivity is greater than the expected sensitivity threshold are repeatedly performed; When the quantization sensitivity is less than or equal to the expected sensitivity threshold, the current preset quantization type is used as the first expected quantization type.

4. The neural network quantization method according to claim 1, characterized in that: The step of obtaining the activation output distribution of the target network layer of the neural network model comprises the following steps: Acquire a calibration data set, wherein the calibration data set includes a plurality of unlabeled data; The calibration data set is input into a neural network model for inference and the output of a target network layer in the neural network model is extracted to obtain an activation output distribution of the target network layer.

5. The neural network quantization method according to claim 1, characterized in that: The data distribution characteristics include variance, skewness and kurtosis, and the analyzing the activation output distribution to determine the data distribution characteristics includes the following steps: Calculate the average of the squares of the differences between each data and the mean in the activation output distribution to obtain the variance; Calculating the square root of the variance to obtain the standard deviation; Calculating the third-order center distance of the activation output distribution, and determining the skewness based on the third-order center distance and the standard deviation; The fourth-order center distance of the activation output distribution is calculated, and the kurtosis is determined based on the fourth-order center distance and the standard deviation.

6. The neural network quantization method according to claim 5, characterized in that: Determining the corresponding quantization strategy according to the data distribution characteristics comprises the following steps: Determine whether the data distribution characteristic meets a first preset condition, wherein the first preset condition is that the skewness is less than a first preset value, the variance is less than a second preset value, and the kurtosis is less than a third preset value; When the data distribution characteristic meets the first preset condition, determining that the quantization method in the quantization strategy is a minimum-maximum-value unsaturated quantization method; When the data distribution feature does not meet the first preset condition, determining whether the data distribution feature meets the second preset condition, wherein the second preset condition is that the skewness is greater than the first preset value or the kurtosis is greater than the third preset value; When the data distribution characteristic satisfies the second preset condition, determining that the quantization method in the quantization strategy is a KL divergence saturation quantization method; When the data distribution feature does not meet the second preset condition, determining whether the data distribution feature meets a third preset condition, wherein the third preset condition is that the variance is greater than or equal to the second preset value; When the data distribution characteristic satisfies the third preset condition, determining that the quantization method in the quantization strategy is an alternating direction multiplier method saturation quantization method; When the data distribution characteristics do not satisfy the third preset condition, it is determined that the quantization method in the quantization strategy is a simple quantization method.

7. The neural network quantization method according to claim 1, characterized in that: Determining the network quantization error according to the expected quantization result comprises the following steps: Performing an inverse quantization process of the original quantization type on the expected quantization result to obtain an inverse quantization result; A network quantization error is determined according to the inverse quantization result and the original output result of the target network layer.

8. A neural network quantization system, characterized in that: include: The first module is used to obtain the activation output distribution of the target network layer of the neural network model; A second module is used to analyze the activation output distribution to determine data distribution characteristics, and determine a corresponding quantization strategy according to the data distribution characteristics, wherein the quantization strategy includes a quantization method; A third module is used to perform a quantization process of a first expected quantization type on the activation output value of the target network layer according to the quantization strategy to obtain an expected quantization result; A fourth module is used to determine a network quantization error according to the expected quantization result; The fifth module is used to determine whether the network quantization error is less than the expected error threshold. When the network quantization error is less than the expected error threshold, the quantization strategy and the first expected quantization type are determined as the quantization parameters of the target network layer to quantize the activation output value of the target network layer of the neural network according to the quantization parameters.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images