Quantization method of neural network model and device for quantizing neural network model
Through the binary approximation method and the minimum mean square variance theory, and combined with the data distribution characteristics, the accuracy and complexity problems of the quantization method in the existing technology under the data asymmetry or unevenness are solved, and the efficient quantization of the neural network model is realized. , significantly compress storage space and improve computing speed.
Patent Information
- Application Number
- CN202011564315.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-25
AI Technical Summary
While compressing the memory space of the neural network model and improving the computing rate, existing quantization methods are difficult to take into account the accuracy and memory usage of the model, especially when the data distribution is asymmetric or uneven, resulting in a decrease in accuracy or an increase in computational complexity.
The binary approximation method is used to calculate the quantization parameters, and the data distribution interval is divided by dimensionality reduction and dividing the data distribution interval, obtain the boundary value of the interval, combine the minimum mean square variance theory and data distribution characteristics, and personalize the quantization factor parameters to reduce the search space of the quantization parameters, and have A mark operator is inserted before the operator of the weight to record the weight quantization information, avoiding the pre-known distribution assumption.
With almost no loss of model prediction accuracy, the storage space is greatly compressed and the computing rate is improved. It is suitable for devices with limited storage space, and maintains high fit and low errors under different data distributions.
Smart Images

Figure CN112580805B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and more specifically, to a quantization method of a neural network model and a device for quantizing a neural network model. Background Art
[0002] With the widespread application of neural network models, the complexity of original neural network models has become increasingly higher, and they cannot run on many devices with limited memory capacity. Therefore, the study of quantization methods for deep learning has long become a research hotspot. For a high-precision neural network model, the higher the proportion of parameters of a certain operator in the total number of parameters of the original model, the higher the frequency of use of the operator. Therefore, quantizing all such high-frequency operators from the original floating-point type to integer type for subsequent operations will greatly improve the memory occupancy and operation rate of the original neural network model with less precision loss, and compress the size of the original model.
[0003] The research goal of neural network model quantization is to find a method that can significantly compress the memory space occupied by the original deep learning model and significantly improve the computing speed of the original model while minimizing the loss of the original model's prediction accuracy. The input of this method is the original high-precision floating-point deep model, and the output is a quantized model that is quantized into a low-precision integer. This has a very important application prospect in practical applications: using the quantized neural network model, the corresponding prediction tasks can be efficiently completed through the original neural network model on many small storage terminals.
[0004] However, existing quantization methods often lead to the problem that the accuracy and memory of the quantized neural network model cannot be achieved at the same time. Summary of the invention
[0005] The purpose of the present invention is to provide a method for quantizing a neural network model and a device for quantizing a neural network model.
[0006] According to one aspect of the present disclosure, a quantization method for a neural network model is provided, the quantization method comprising: obtaining a neural network model; calculating a quantization parameter corresponding to an operator to be quantized of the neural network model based on a bisection approximation method; and quantizing the operator to be quantized of the neural network model based on the quantization parameter to obtain a neural network model having a quantized operator to be quantized.
[0007] Optionally, the step of calculating the quantization parameter corresponding to the operator to be quantized of the neural network model includes: using a verification data set to verify the neural network model to obtain input data of each operator to be quantized; using a bisection approximation method to calculate the quantization parameter corresponding to the minimum mean square difference of the input data of each operator to be quantized before and after quantization based on the input data of each operator to be quantized.
[0008] Optionally, the step of calculating the quantization parameter corresponding to the minimum mean square difference includes: reducing the dimension of the input data of each operator to be quantized; dividing the input data of each operator to be quantized after the dimension reduction into multiple data distribution intervals based on the statistical characteristics of the input data of each operator to be quantized after the dimension reduction, and obtaining an interval supremum value array, wherein the interval supremum value array records the supremum value of each data distribution interval; using a bisection approximation method to perform bisection approximation toward the middle between the starting point and the end point of each data distribution interval to search for the quantization parameter corresponding to the minimum mean square difference.
[0009] Optionally, the quantization parameter includes: at least one of a truncation parameter of a data distribution interval, a quantization factor parameter and a truncation factor parameter.
[0010] Optionally, the step of searching for quantization parameters includes: for each data distribution interval, when obtaining the supremum value array of the interval, initializing the minimum average square difference to the initial average square difference of the data distribution interval, wherein the initial average square difference corresponds to the quantization parameter corresponding to the midpoint between the starting point and the end point of the data distribution interval; by binary approximation to the middle between the starting point and the end point of the data distribution interval, calculating the average square difference of the approaching point of the data distribution interval, wherein the average square difference of the approaching point corresponds to the quantization parameter corresponding to the approaching point of the data distribution interval; when the average square difference of the approaching point is less than the minimum average square difference, using the average square difference of the approaching point to update the minimum average square difference; when the data distribution interval is traversed, outputting the quantization parameter corresponding to the minimum average square difference as the quantization parameter.
[0011] Optionally, the operators to be quantized of the neural network model include quantizable operators of the neural network model, wherein, in response to the ratio of parameters contained in the operators of the neural network model to total parameters in the neural network exceeding a threshold, or being computationally intensive operators, the operators are quantizable operators.
[0012] Optionally, the quantization method also includes: before calculating the quantization parameter corresponding to the quantizable operator, marking the quantizable operator by inserting a quantization marking operator before the quantizable operator, wherein the step of marking the quantizable operator includes: determining whether the input data of the quantizable operator has weight data; when the input data of the quantizable operator does not have weight data, inserting the quantization marking operator before the quantizable operator; when the input data of the quantizable operator has weight data, inserting the quantization marking operator before the quantizable operator, and inserting the quantization marking operator before the weight data to mark whether the weight data needs to be quantized.
[0013] Optionally, the neural network model is a deep learning neural network model trained to perform one of image recognition, natural language processing, and recommendation system processing.
[0014] According to one aspect of the present disclosure, there is provided a device for quantizing a neural network model, the device comprising: a data acquisition module configured to acquire a neural network model; a quantization parameter calculation module configured to calculate, based on a bisection approximation method, quantization parameters corresponding to operators to be quantized of the neural network model; and a quantization implementation module configured to quantize the operators to be quantized of the neural network model based on the quantization parameters to obtain a neural network model having quantized operators to be quantized.
[0015] Optionally, the quantization parameter calculation module is configured to: use a verification data set to verify the neural network model to obtain the input data of each operator to be quantized; use a bisection approximation method to calculate the quantization parameter corresponding to the minimum mean square difference of the input data of each operator to be quantized before and after quantization from the input data of each operator to be quantized.
[0016] Optionally, the quantization parameter calculation module is configured to: reduce the dimension of the input data of each operator to be quantized; based on the statistical characteristics of the input data of each operator to be quantized after the dimension reduction, divide the input data of each operator to be quantized after the dimension reduction into multiple data distribution intervals, and obtain an interval supremum value array, wherein the interval supremum value array records the supremum value of each data distribution interval; use the bisection approximation method to perform bisection approximation toward the middle between the starting point and the end point of each data distribution interval to search for the quantization parameter corresponding to the minimum mean square difference.
[0017] Optionally, the quantization parameter includes: at least one of a truncation parameter of a data distribution interval, a quantization factor parameter and a truncation factor parameter.
[0018] Optionally, the quantization parameter calculation module is configured as follows: for each data distribution interval, when obtaining the supremum value array of the interval, the minimum average square difference is initialized to the initial average square difference of the data distribution interval, wherein the initial average square difference corresponds to the quantization parameter corresponding to the midpoint between the starting point and the end point of the data distribution interval; by bisection approximation to the middle between the starting point and the end point of the data distribution interval, the approach point average square difference of the data distribution interval is calculated, wherein the approach point average square difference corresponds to the quantization parameter corresponding to the approach point of the data distribution interval; when the approach point average square difference is less than the minimum average square difference, the approach point average square difference is used to update the minimum average square difference; when the data distribution interval is traversed, the quantization parameter corresponding to the minimum average square difference is output as the quantization parameter.
[0019] Optionally, the operators to be quantized of the neural network model include quantifiable operators of the neural network model, wherein, when the proportion of parameters contained in the operators of the neural network model to the total parameters in the neural network exceeds a threshold, or is a computationally intensive operator, the operator is a quantifiable operator.
[0020] Optionally, the device also includes: a quantization marking module, configured to: mark the quantizable operator by inserting a quantization marking operator before the quantizable operator before calculating the quantization parameter corresponding to the quantizable operator, wherein the quantization marking module is configured to: determine whether the input data of the quantizable operator has weight data; when the input data of the quantizable operator does not have weight data, insert the quantization marking operator before the quantizable operator; when the input data of the quantizable operator has weight data, insert the quantization marking operator before the quantizable operator, and insert the quantization marking operator before the weight data to mark whether the weight data needs to be quantized.
[0021] Optionally, the neural network model may be a deep learning neural network model trained to perform one of image recognition, natural language processing, and recommendation system processing.
[0022] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by one or more computing devices, the one or more computing devices implement any of the quantization methods described above.
[0023] According to one aspect of the present disclosure, there is provided a multi-round human-computer dialogue system comprising one or more computing devices and one or more storage devices, wherein the one or more storage devices have recorded thereon a computer program, and when the computer program is executed by the one or more computing devices, the one or more computing devices implement any of the quantization methods described above.
[0024] The quantization method of the present invention can effectively reduce the search space of the quantization parameter by calculating the quantization parameter corresponding to the operator to be quantized based on the binary approximation method, find the optimal quantization parameter, and obtain the quantized neural network model. Compared with the neural network model before quantization, the quantized neural network model can occupy relatively less storage space, thereby improving the utilization efficiency of the memory, and saving the central processing unit, image processor and / or neural processor with relatively small overhead, and the performance will not be significantly degraded.
[0025] In addition, the quantization method of the present invention can record the subsequent weight quantization parameter information of the operator by adding a marking operator before the weighted operator to be quantized. Therefore, the quantization of the weighted operator can be completed without the need for known prior distribution of weight data, thereby reducing the difficulty of information acquisition of the original model quantization method.
[0026] In addition, in the present invention, the quantization error of each data in the distribution subinterval of the current quantized data (i.e., the mean square error of the quantized data and the data before quantization) is calculated to replace the solution process based on the cross entropy theory in the existing method. The quantization error calculated in this way can better reflect the data distribution characteristics when calculating the asymmetric data distribution compared with the cross entropy theory, and has a higher fit with the model, thereby overcoming the following problem: in the solution process based on the cross entropy theory in the existing method, its quantization parameter cannot include most of the positive data, resulting in the truncation of a large amount of valid positive data.
[0027] In addition, the present invention proposes a method for combining data distribution sub-intervals with minimum mean square error to individually calculate quantization factors, which can effectively overcome the quantization problem of uneven data distribution.
[0028] In addition, the quantization method of the present invention combines the statistical distribution characteristics of different original input data, and can calculate the optimal quantization factor parameter set adapted to the distribution of each input data in an individualized manner, and select the optimal quantization factor parameter applicable to the entire data through the minimum mean square error theory. Therefore, regardless of whether the input data is symmetrically distributed or asymmetrically distributed, the optimal quantization factor parameter can be calculated by this method and the model quantization can be performed.
[0029] In addition, the present invention proposes to use a binary approximation method to reduce the search space of the original quantization parameters to be traversed according to the relaxed monotonic change rule of the mean square error, thereby greatly improving the time for solving the quantization parameters.
[0030] In addition, the present invention proposes to determine the truncation parameter by combining the mean square error theory and data distribution characteristics to calculate the mean square error of the supremum value of each interval according to the divided intervals, which can include a large amount of positive half-axis data while also ensuring that the error of the input data of this layer is minimized after quantization, and ultimately will not lead to a significant decrease in the accuracy of the overall quantization model.
[0031] In addition, the quantization method of the present invention can greatly improve the operation rate of the original model without losing the prediction accuracy of the model, and greatly compress the storage space occupied by the original model; and the original model quantized by this method can be compatible with the back-end such as CPU and GPU. This enhances the applicability of the original deep model for many small devices with limited storage space, and has broad application prospects in more hardware terminals. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The above and other objects and features of the present invention will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate an example, in which:
[0033] Figure 1Distribution diagrams showing common asymmetric or uneven input data;
[0034] Figure 2 A flowchart showing a quantization method of a neural network model according to an exemplary embodiment of the present disclosure;
[0035] Figure 3 An example showing MSE relaxing a monotonic trend;
[0036] Figure 4 A schematic diagram of a binary approximation method according to an embodiment of the present invention is shown;
[0037] Figure 5 An apparatus for quantizing a neural network model according to an exemplary embodiment of the present invention is shown;
[0038] Figure 6 A structural diagram showing a quantization method of a neural network model according to an exemplary embodiment of the present invention;
[0039] Figure 7 A schematic diagram showing quantization factor parameters calculated by a quantization method of a neural network model according to an exemplary embodiment of the present invention and quantization factor parameters calculated by an existing quantization method;
[0040] Figure 8 A detailed flow chart of a quantization method of a neural network model according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION
[0041] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to assist in understanding, but these details are considered to be exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures are omitted for clarity and brevity.
[0042] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0043] The existing quantization methods for deep models mainly include deterministic quantization methods and random quantization methods. The existing quantization method based on deterministic truncation function mainly uses the truncation function to convert continuous values (i.e. high-precision floating-point numbers) into discrete values (i.e. low-precision integers) to complete model quantization. The existing truncation function setting method is to set a truncation function in advance according to the global data distribution, or use the maximum value as the truncation value. In more cases, a truncation parameter determined based on the cross-entropy theory is used for quantization. However, in the existing methods, using the maximum value as the truncation value for quantization will result in a significant decrease in accuracy. In order to improve the accuracy, the quantized model must be trained. The determination of the truncation parameter based on the cross-entropy theory will only have better accuracy when the distribution is relatively symmetrical and uniform. In some cases where the data distribution is uneven and asymmetric, such as Figure 1 As shown in Figure 2, the accuracy will drop significantly after quantization. Figure 1 Represents a common asymmetric or uneven input data distribution graph. Figure 1 It can be seen that a large number of negative numbers are concentrated near 0, and positive numbers are scattered in the distribution range from 0 to 4000 on the positive semi-axis.
[0044] The existing vectorization-based methods mainly cluster the original high-precision operators into subgroups and then quantize them by subgroups. The existing vectorization methods mainly use the K-Means-based clustering method. Although this method is highly operable, it can only be quantized on a pre-trained model with a defined truncation function, so the versatility of this method is poor.
[0045] Existing random quantization methods are mainly divided into quantization methods based on random truncation and quantization methods based on probability distribution. The quantization method based on random truncation mainly provides a method to inject noise during the training process, which can act as a regularizer and enable conditional calculations. However, this method requires the distribution characteristics of the noise data to be known, so the practicality of this method is poor. The quantization method based on probability distribution requires the assumption that the weight data is a discrete distribution, and also requires prior knowledge of the distribution of the known weight data. However, the acquisition of the prior distribution knowledge of the weights is often difficult to achieve in practical applications, so the versatility of this method is very limited.
[0046] Existing quantization methods always lead to a significant decrease in the prediction performance of the original model due to the following problems:
[0047] 1. The existing maximum value truncation quantization method may cause the performance of the original deep model to drop sharply after each truncation operation. To improve the accuracy, the quantized model needs to be trained again. At the same time, since the parameter space of the quantized model is small when it is calculated using truncated discrete values, it is difficult for the quantized model to converge during the training process. In addition, the truncation operation cannot utilize the structural information of the weights in the network. And the quantized model still needs the original accuracy real value to assist in training, which is very time-consuming.
[0048] 2. The quantitative method based on the cross entropy theory to determine the truncation parameter is not very effective when the distribution characteristics of the original input data are rare or asymmetric (such as Figure 1 As shown in the figure, the optimal quantization parameters cannot be calculated, which causes the prediction performance of the original model to be affected when non-optimal quantization parameters are used for calculation.
[0049] 3. The vector quantization method based on K-means clustering has a large amount of calculation. Compared with the truncated quantization method, it is difficult to get integer weights with the vector quantization method. Vector quantization is usually used to quantize pre-trained models. Therefore, if the task is to train a quantized deep model from scratch, a pre-set truncation function is required, which is difficult to set in advance in practical applications.
[0050] 4. Random truncation quantization method uses random rounding truncation quantization method, which requires parameter estimation of many intermediate parameters, and this estimation often has a large deviation. This deviation may cause the oscillation of the loss function during training, which in turn sharply affects the prediction performance of the model.
[0051] 5. Probability-based quantization methods require a predefined appropriate weight prior distribution, but it is often difficult to find a suitable weight distribution in advance for the model. At the same time, many quantization methods need to traverse a large solution space when solving the optimal parameters due to limited prior knowledge, and the calculation process is quite time-consuming.
[0052] In order to solve one or more of the problems existing in the above-mentioned existing technologies, taking into account both accuracy and memory, the present disclosure provides a quantization method for a neural network model using a binary approximation method. According to the quantization method for a neural network model using a binary approximation method of the present invention, compared with the neural network model before quantization, the quantized neural network model can occupy relatively less storage space, thereby improving the utilization efficiency of the memory. In addition, when the quantized neural network model is executed by a central processing unit, an image processor, and / or a neural processor in an electronic device (e.g., a mobile device) to perform tasks such as recognition, the central processing unit, the image processor, and / or the neural processor can perform corresponding calculations based on the quantized neural network model with relatively small overhead without affecting the accuracy of tasks such as recognition. That is, in the present invention, the quantized neural network model obtained by the quantization method of the neural network model using the binary approximation method can improve the hardware performance of the electronic device (e.g., improve the utilization of memory and / or reduce the overhead of the central processing unit, the image processor, and / or the neural processor) compared with the neural network model before quantization.
[0053] Figure 2 A flowchart of a quantization method of a neural network model according to an exemplary embodiment of the present disclosure is shown.
[0054] Reference Figure 2 , in step S210, a neural network model is obtained.
[0055] Here, the neural network model may be a pre-trained neural network model. For example, the neural network model may be a deep learning model whose original precision is a floating point number. In one example, the neural network model may be obtained from a database, which may be a database of a server (e.g., a cloud server) or may be a database of a mobile device with limited memory. However, the present invention is not limited to obtaining a neural network model from a database, and it is also feasible to obtain a neural network model from any other hardware device. In one embodiment, the neural network model may be a deep learning neural network model trained to perform one of image recognition, natural language processing, and recommendation system processing.
[0056] In step S220, a quantization parameter corresponding to the operator to be quantized is calculated based on a binary approximation method. In the present invention, preferably, the binary approximation method is a bidirectional binary approximation method.
[0057] In the present invention, by calculating the quantization parameter corresponding to the operator to be quantized based on the binary approximation method, the search space of the quantization parameter can be effectively narrowed down and the quantization parameter can be found at the same time.
[0058] For example, the operators to be quantized of the neural network model may include quantizable operators of the neural network model. In one example, when the proportion of parameters contained in the operators of the neural network model to the total parameters in the neural network exceeds a threshold, or is a computationally intensive operator, the operator is a quantizable operator. A computationally intensive operator may be an operator that includes a large number of matrix multiplication operations (for example, as an example only, a convolution operator, a fully connected layer operator, etc.).
[0059] According to an exemplary embodiment of the present invention, before calculating the quantization parameter corresponding to the quantizable operator, the quantizable operator may be marked by inserting a quantization marking operator before the quantizable operator. In other words, the operator is marked as requiring quantization by inserting the quantization marking operator. For example, it may be determined whether the input data of the quantizable operator has weight data. When the input data of the quantizable operator does not have weight data, the quantization marking operator may be inserted before the quantizable operator. When the input data of the quantizable operator has weight data, the quantization marking operator may be inserted before the quantizable operator, and the quantization marking operator may be inserted before the weight data to mark whether the weight data needs to be quantized.
[0060] In the present invention, a marking operator is added before the weighted operator to be quantized, thereby recording the subsequent weight quantization parameter information of the operator, so that the quantization of the weighted operator can be completed without the need for the prior distribution of the weight data in advance. In other words, there is no need for a predefined truncation function and weight prior distribution information, and the optimal quantization parameter set suitable for this model is calculated based on the original input data in a personalized manner, which reduces the difficulty of obtaining information for the original model quantization method.
[0061] That is to say, if it is a quantizable operator, it is determined whether the input data has weight data. If it has weight data, a quantization marking operator (2 in total) is inserted to mark whether the weight data needs to be quantized; if it does not have weight data, the quantization marking is completed and the next operator is processed. FP Q N All labeling operations are completed, among which the deep learning model with original floating point precision can be used with D FP =(Q N , O M ) indicates that, among which, Q N Represents the set of operators that need to be quantized in the model, where N is the number of operators that need to be quantized; M It represents the set of operators that do not need to be quantized in the model, and M is the number of operators that do not need to be quantized.
[0062] In one embodiment of the present invention, a verification data set can be used to verify the neural network model to obtain the input data of each operator to be quantized. Then, the bisection approximation method is used to calculate the quantization parameter corresponding to the minimum mean square difference of the input data of each operator to be quantized before and after quantization from the input data of each operator to be quantized. In the present invention, the quantization error of each data in the distribution subinterval of the data after the current quantization (i.e., the mean square error of the data after quantization and the data before quantization) is calculated to replace the solution process based on the cross entropy theory in the existing method. The quantization error calculated in this way can better reflect the data distribution characteristics when calculating the asymmetric data distribution than the cross entropy theory, and has a higher fit with the model. Because, in the solution process based on the cross entropy theory in the existing method, the quantization parameter cannot contain most of the positive data, resulting in the truncation of a large amount of valid positive data.
[0063] Here, in order to better use the binary approximation method to process the input data of the operator to be quantized, the input data of each operator to be quantized can be reduced in dimension. R After being injected into the neural network model, the input data of each operator to be quantized is extracted and dimensionality reduction is performed on it, converting the original input data from a multidimensional matrix into a one-dimensional array X i , i∈[0,N) where N is the number of operators to be quantized, which is convenient for dividing the interval and determining the supremum value of each interval. R It is a collection of a small amount of data randomly extracted from the test set of the predictive model.
[0064] In addition, based on the statistical characteristics of the input data of each operator to be quantized after dimensionality reduction, the input data of each operator to be quantized after dimensionality reduction can be divided into multiple data distribution intervals, and an interval supremum value array can be obtained, wherein the interval supremum value array records the supremum value of each data distribution interval.
[0065] Specifically, the input data of each operator to be quantized is divided into intervals. The purpose is to record the supremum value of each interval. The method flow can be seen in the following Algorithm 1:
[0066]
[0067]
[0068] Referring to the above algorithm 1, first extract each operator input data set X to be quantized i The statistical characteristic distribution information P(X i ), mainly considering the minimum value and maximum value Determine the quantization interval threshold thres according to the absolute value of the extreme value:
[0069]
[0070] Then, divide the number of intervals n according to thres and known data distribution bins , calculate the length of each interval inc:
[0071]
[0072] Among them, n bins =8001 is a generalizable setting value, which indicates the number of data distribution intervals in which the original input data of each operator to be quantized is quantized from the original floating point precision to the integer precision.
[0073] Finally, use the interval supremum value array Record the supremum value of each data distribution interval. The supremum value of the jth subinterval is T j :
[0074] T j =-thres+(inc×j),j∈[0,n bins ].
[0075] After dividing the data distribution intervals, the bisection approximation method can be used to bisection approximate the middle between the starting point and the end point of each data distribution interval to search for the quantization parameter corresponding to the minimum mean square error. Generally, the existing quantization methods cannot select the best quantization parameter suitable for the overall data when the original input data of the operator to be quantized is asymmetric or uneven. Therefore, the present invention proposes a method for combining the data distribution sub-intervals with the minimum mean square error to calculate the quantization factor in a personalized way, which can effectively overcome the quantization problem of uneven data distribution.
[0076] Here, the quantization method of the present invention combines the statistical distribution characteristics of different original input data, and can calculate the optimal quantization factor parameter set adapted to the distribution of each input data in an individualized manner, and select the optimal quantization factor parameter applicable to the entire data through the minimum mean square error theory. Therefore, regardless of whether the input data is symmetrically distributed or asymmetrically distributed, the optimal quantization factor parameter can be calculated by the quantization method of the present invention, and model quantization can be performed.
[0077] In the present invention, the quantization parameter may include at least one of a truncation parameter of a data distribution interval, a quantization factor parameter, and a truncation factor parameter.
[0078] The following uses the truncation parameter α as an example to describe in detail how to use the bisection approximation method.
[0079] In the present invention, the subscript of the segment interval corresponding to the optimal truncation parameter corresponding to the minimum mean square difference is quickly searched by using the bisection approximation method. Since the trend of the mean square difference is a relaxed monotonically decreasing trend, as shown in the example of the MSE relaxed monotonically decreasing trend Figure 3 As shown, its overall change trend is a decreasing curve, but when the truncation parameter α is very large, its curve will have a slightly rising interval. Therefore, the present invention uses the bisection approximation method to continuously bisection the minimum MSE from the two endpoints to the middle to finally obtain the optimal truncation parameter α.
[0080] Figure 4 A schematic diagram of a binary approximation method according to an embodiment of the present invention is shown.
[0081] In the present invention, in order to solve the problems of large redundant calculation amount and long quantization time in the existing methods when traversing the search space of truncation factors, the present invention proposes to use the bisection approximation method to narrow the search space of the original quantization parameters to be traversed according to the relaxed monotonic change law of the mean square error, thereby greatly improving the time for solving the quantization parameters.
[0082] The optimal truncation parameter α can be calculated by the bisection approximation method to reduce , find the optimal truncation parameter α corresponding to the minimum mean square difference before and after quantization in the search space, and prepare for the quantization implementation module. In one embodiment of the present invention, in order to calculate the minimum mean square difference, for each data distribution interval, when obtaining the interval supremum value array, the minimum mean square difference can be initialized to the initial mean square difference of the data distribution interval, wherein the initial mean square difference corresponds to the quantization parameter corresponding to the midpoint between the starting point and the end point of the data distribution interval. Then, by approximating the middle between the starting point and the end point of the data distribution interval, the average square difference of the approaching point of the data distribution interval is calculated, wherein the average square difference of the approaching point corresponds to the quantization parameter corresponding to the approaching point of the data distribution interval. When the average square difference of the approaching point is less than the minimum average square difference, the average square difference of the approaching point is used to update the minimum average square difference. For the specific method flow, please refer to Algorithm 2 below.
[0083]
[0084]
[0085] Referring to Algorithm 2, the step of calculating the optimal truncation parameter using the bisection approximation method may include:
[0086] 1. Initialize the traversal start and end subscripts. First calculate the traversal start p and end q of the starting interval:
[0087]
[0088] q=nbins
[0089] Here, bit represents the number of integer bits after quantization, which is generally an integer power of 2 (such as 8, 16, etc.).
[0090] Calculates the midpoint between the start and end subscripts.
[0091] m=(p+q) / 2
[0092] 2. Since the trend of the mean square error is a relaxed monotonically decreasing trend, the subscript of the optimal truncation variable must be somewhere in the middle (such as Figure 3 As shown in Figure 2.2, p is moved closer to the middle subscript m in steps of two until the calculated MSF is P ≤MSE m Then the subscript p goes back one step and jumps out of the loop.
[0093] Keep moving q closer to the middle subscript m in steps of two until MSE q ≤MSE m Then the subscript q goes back one step and jumps out of the loop.
[0094] Among them, the forward step length of p is calculated as: p = (p + m) / 2, and the backward step length of p is: p = (p * 2) - m
[0095] Similarly, calculate the forward step length of q: q = (q + m) / 2, and the backward step length of q: q = (q * 2) - m
[0096] 3. Traverse the supremum value array of the partition interval from p to q And calculate the quantization factor parameters α, scale, clip of the current sub-interval according to the following formula min , clip max And the minimum mean square error MSE. For the specific method flow, please see Algorithm 3 (Method for calculating the mean square error MSE) below.
[0097] α=T j , j∈[p,q]
[0098]
[0099] clip min =-(2 bit-1 ), clip max =2 bit-1
[0100]
[0101] X′=Q×scale
[0102]
[0103] Where bit is the integer number of digits of the model accuracy after quantization. Clip(.) represents the truncation function, and the specific calculation formula is:
[0104]
[0105] When the traversal is completed, the truncation parameter subscript corresponding to the minimum mean square difference is obtained, and the optimal truncation parameter α = T is returned. j , where MSE j is the minimum value.
[0106] In the method for calculating the mean square difference (Algorithm 3), the search space is traversed and the mean square difference between the value after dequantization of the input data using the current truncation parameter and the original value is calculated. Whether the truncation parameter is optimal is determined based on whether the value is the global minimum, thereby determining whether the current quantization factor parameter is optimal.
[0107]
[0108]
[0109] In step S230, the operator to be quantized is quantized based on the quantization parameter to obtain a neural network model with the quantized operator to be quantized.
[0110] In one embodiment of the present invention, the optimal truncation parameter α, the optimal quantization factor scale and the corresponding clip can be used. min and clip max , realize the specific quantization operation and dequantization operation of the operator to be quantized.
[0111] Specifically, first, the optimal quantization factor and truncation parameter set are calculated
[0112]
[0113] clip min =-(2 bit-1 ), clip max =2 bit-1 ,
[0114] And the optimal scale and clip min , clip max Assign to the quantization mark operator before each operator to be quantized.
[0115] Then, all quantization mark operators in the quantizable model are converted into the underlying operation mode (multiplication, rounding, truncation, conversion type) through the underlying hardware (such as GPU, NPU, etc.), and the integer quantization of each operator to be quantized is realized using the optimal quantization factor parameter scale. And the dequantization operation is realized after each operator to be quantized is operated.
[0116] Finally, the output is the original depth model D with a quantized precision of bit integer INT .
[0117] Figure 5 An apparatus for quantizing a neural network model according to an exemplary embodiment of the present invention is shown.
[0118] Reference Figure 5 , the device 500 for quantizing a neural network model may include a data acquisition module 510, a quantization parameter calculation module 520, and a quantization implementation module 540. Here, the data acquisition module 510 is configured to acquire a neural network model. The quantization parameter calculation module 520 is configured to calculate a quantization parameter corresponding to a quantizable operator based on a binary approximation method. The quantization implementation module 530 is configured to quantize the operator to be quantized based on the quantization parameter to obtain a neural network model with a quantized operator to be quantized.
[0119] In other words, the data acquisition module 510 performs the data acquisition operation, the quantization parameter calculation module 520 performs the quantization parameter calculation operation, and the quantization implementation module 540 performs the quantization operation. Figure 2 The operations of obtaining data operations to calculate quantization parameters, quantization operations, and database update operations are specifically described, so the same description is not repeated here. In addition, optionally, the device 500 for quantizing the neural network model may include a quantization marking module (not shown) configured to mark the operator to be quantized.
[0120] Figure 6 A structural diagram showing a quantization method of a neural network model according to an exemplary embodiment of the present invention.
[0121] Reference Figure 6 , the input may correspond to the acquired neural network model (i.e., the original model). In the quantization marking module, a quantizable original model is generated by inserting a quantization operator into the original model. Specifically, the quantization marking module identifies and marks each quantizable operator in the original input model for calculation and use in the quantization implementation. The module finally outputs an original floating-point precision model with a quantization marking operator. Traverse each operator of the neural network model, select all operators that need to be quantized, and insert a quantization marking operator (simulated_quantize) before these operators to implement the quantization operation in the subsequent quantization implementation module.
[0122] exist Figure 6 In the quantization marking module, the quantization parameter calculation module can provide the quantizable original model to the verification calculation module (also called the quantization parameter calculation module). The verification calculation module is designed to calculate the quantization parameter set of the original model, and then complete the quantization calculation of the original model in the quantization implementation module. In the original model of the existing quantization marking operator, the statistical distribution characteristics of the original input data are first obtained through the verification data set, and then the quantization parameters are calculated based on the theory of minimum mean square error (MSE) using the bisection approximation method: quantization factor scale, truncation factor parameter clip min and clip max , as the output of this module, is used by the subsequent quantization implementation module to perform quantization calculations using this parameter set. Figure 6 In the answer box, N means no and Y means yes.
[0123] Reference Figure 6 ,The quantization implementation module mainly completes the quantization and dequantization calculation functions and outputs the final quantization model. After obtaining the output of the previous module, the quantization parameter group is assigned to the quantization marking operator, which implements the quantization operation (such as multiplication, rounding, truncation, data type conversion, etc.) in the underlying hardware. Finally, the quantized integer precision model corresponding to the input original depth model is obtained.
[0124] Figure 7 A schematic diagram showing quantization factor parameters calculated by a quantization method of a neural network model according to an exemplary embodiment of the present invention and quantization factor parameters calculated by an existing quantization method.
[0125] The existing quantization method determines the truncation parameter α according to the predefined truncation function of the data distribution before and after quantization, which will cause a large amount of positive semi-axis data to be truncated, and ultimately cause a significant decrease in the overall accuracy of the model after quantization. However, if the method proposed by the present invention is used to determine the truncation parameter α by combining the mean square error theory and the data distribution characteristics and calculating the mean square error of the supremum value of each interval according to the divided interval, a large amount of positive semi-axis data can be included, and at the same time, the error of the input data of this layer after quantization is minimized, which will ultimately not cause a significant decrease in the accuracy of the overall quantization model.
[0126] In order to test the effect of the quantization method of the present invention, multiple test number sets are used to test the quantization method of the present invention and the existing method. The test results are shown in Table 1 below.
[0127] Table 1. Test results
[0128]
[0129] Among them, SQuAD1.1 is the test data set of the BERT model, which uses F1 as the evaluation indicator; ML-1M is the test data set of the NCF model, which uses HQ@10 as the evaluation indicator;
[0130] Referring to Table 1, in one example, taking the natural language processing model BERT as an example, the quantization method of the present invention realizes the quantization of the original model, and the prediction accuracy of the model quantized to 8-bit integer precision (ie, bit=8) is not significantly reduced compared with the original model (32-bit floating point precision), and the prediction performance is greatly improved compared with the existing quantization effect based on cross entropy theory. The quantization method of the present invention compresses the model size to 1 / 4 of the original, and the operation speed of the quantized model is 2.23 times that of the original model.
[0131] In another example, taking the NCF model of the recommendation system as an example, the weight data is quantized using the quantization method of the present invention, and quantization can also be achieved when the weight distribution information is unknown. The model accuracy does not decrease significantly after quantization, and the accuracy is slightly higher than the quantization accuracy of the weight quantization based on the cross entropy theory. The model size is compressed to 1 / 4 of the original, and the model operation speed does not slow down after quantization. Therefore, when the weight distribution is unknown, the quantization method of the present invention can also complete quantization well on the model of the recommendation system.
[0132] In another example, taking the image recognition model ResNet-50 as an example, the model accuracy does not drop significantly by using the quantization of the present invention, which is slightly better than the quantization accuracy based on the existing cross entropy method. The model size is compressed to 1 / 4 of the original model. The operation speed of the quantized model is 2.577 times that of the original model.
[0133] It can be seen from Table 1 above that the quantization method of the present invention can greatly improve the operation rate of the original model without losing the prediction accuracy of the model, and greatly compress the storage space occupied by the original model; and the original model quantized by the quantization method of the present invention can be compatible with the back-end such as CPU and GPU. This enhances the applicability of the original deep model for many small devices with limited storage space, and has broad application prospects in more hardware terminals.
[0134] For example, when using a neural network to perform image recognition learning tasks, take the ReNet-50 network as an example: the model contains a large number of Conv2d operators, so it is very suitable for application in image-related tasks. The picture of the picture data set is first converted from pixels to lattice vectors, and enters a pre-classifier (generally a basic classifier, such as a logistic regression classifier, a Bayesian classifier, etc.) for training, and a trained model is obtained by minimizing the error value of the loss function during training. However, at this time, the data types of the model during calculation are all floating point types, and the model storage requires a lot of space overhead, so it is necessary to quantize the convolution operator of the model so as to complete the compression storage of the model without affecting the prediction accuracy of the model. Through the quantization method of the present invention, the size of the model can be greatly reduced while maintaining the original accuracy of the ResNet-50 model. The truncation factor and quantization parameter quickly calculated by the bidirectional binary approximation method can accurately map high-precision floating-point input data to a low-precision integer range to become integer data, and then pass the integer input data to terminals such as the CPU, GPU or multi-core NPU for data reading, which can greatly shorten the model's prediction time for each image data and ensure that its prediction accuracy is almost unaffected.
[0135] For example, when using a neural network to perform a natural language processing task, take the Bert-Large model as an example: the original text data is first converted into a word vector for storage through the word2vec toolkit, and then enters the Bert-Large model with a bidirectional transformer to prepare for training. However, the size of the model will still consume too much memory, so quantization is also required. In this model, the DENSE operator has the highest proportion and the operator size is also the largest, so after the DENSE operator is quantized using the method of the present invention, the size of the original model can be compressed to one-fourth of the original size. Similarly, after the text data is converted into a vector via a text converter and quantized by the present invention, the generated integer model can be stored on a device such as a CPU for hardware acceleration, and then read out for tasks such as inference and prediction.
[0136] For example, when using a neural network to build a recommendation system, take the NCF model as an example: the original user-item tuple data enters the NCF for collaborative filtering and obtains the current user preference project ranking, and then selects the project that the user is most interested in according to the Top-K ranking and outputs it as a recommended project. Taking the movielens data set as an example, the user-movie data enters the NCF model, and similarity and user preference calculations are performed. The calculation process is mainly completed by the TAKE operator in the model, and the operator is very large. Therefore, after quantization by the quantization algorithm of the present invention, a quantized model with only a quarter of the size of the original model is obtained. The data enters the quantized model for inference, and this process can also call the back-end accelerators such as CPU and GPU to speed up the calculation. However, the recommendation accuracy will not be greatly affected, and the recommendation calculation time can be reduced. It can be seen that the quantization method of the present invention plays a vital role in the actual task execution process of various types of neural network models.
[0137] Figure 8 A detailed flow chart of a quantization method of a neural network model according to an exemplary embodiment of the present invention is shown.
[0138] Reference Figure 8 The quantization method of the neural network model of the present invention aims to convert the neural network model D whose original precision is a floating point number into FP Model D quantized to integer precision INT , thereby compressing the memory space occupied by the original model and improving the computing speed of the original model with minimal loss in overall prediction accuracy.
[0139] Combination Figure 8 , when input D FP After that, the quantization marking module is used to identify each operator in the model that needs to be quantized and add a quantization marking operator to complete the subsequent quantization calculation. This operation is not performed for operators that do not need to be quantized. After this module is completed, a marked model with a marked operator will be output, and then the verification calculation will begin.
[0140] Verify the calculation module in the marked model D with the marking operator FP The statistical distribution characteristic information of the input data of each operator to be quantized is obtained by verifying the data set; based on the statistical distribution information, the quantization interval threshold thres can be determined, which reflects the distribution information of the data after quantization and ensures that the distribution of the original data remains unchanged during quantization; then thres can be used to calculate the interval length inc of the data distribution interval after quantization; based on inc and the number of divisions n of the data distribution interval after quantization bins The supremum value T of each subinterval can be determined j; Then start to use the bisection approximation method to narrow the search space and calculate the quantization factor truncation parameter α of the current jth interval j , the scale of the current sub-interval j , and the truncation factor parameter clip min and clip max , using the currently calculated quantization parameter scale j Quantize the i-th data X i , get the quantized data Q i , and then Q i Generate X by dequantization i ′, then through X i ′ and X i Calculate the mean square error MSE of the quantization in the current j-th subinterval j , and then sum up with the current minimum MSE min Compare and update, so as to output the best quantization parameters scale and clip after traversing all sub-intervals min and clip max .
[0141] The quantization implementation module assigns the optimal quantization parameters to the labeling operator, and then implements quantization operations (such as rounding, type conversion, multiplication, etc.) through the underlying hardware, and finally outputs an original deep learning model D with integer precision. INT .
[0142] According to an exemplary embodiment of the present disclosure, a computing device is provided, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, a quantization method of a neural network model according to an exemplary embodiment of the present disclosure is implemented.
[0143] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein, when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform a quantization method of a neural network model according to an exemplary embodiment of the present disclosure.
[0144] According to the quantization method and device of the neural network model disclosed in the present invention, by calculating the quantization parameter corresponding to the quantizable operator based on the bisection approximation method, the search space of the quantization parameter can be effectively narrowed down and the quantization parameter can be found at the same time.
[0145] In addition, according to the quantization method and device of the neural network model disclosed in the present invention, the truncation parameter value is set personalized under the condition of very little loss of precision, and there is no need to store the high-precision true value of the original model, thereby reducing memory overhead and compressing the original depth model to about 1 / 4 of the original size, so as to solve the problem that precision and memory cannot be achieved at the same time brought about by the existing truncation quantization method.
[0146] In addition, according to the quantization method and device of the neural network model disclosed in the present invention, the operation rate of the model is improved, and a quantization marking operator is added when processing weighted operators to be quantized to solve the problem of requiring predefined weight data distribution in existing probability-based quantization methods.
[0147] In addition, according to the quantization method and device of the neural network model disclosed in the present invention, according to the relaxed monotonic change law of the mean square error, the bisection approximation method is used to traverse the subintervals of the quantized data distribution, and then the quantization parameters are solved. Compared with the process of traversing each subinterval in the existing method, the quantization method of the present invention reduces the search space of the truncation factor to be traversed and greatly improves the quantization rate.
[0148] In addition, according to the quantization method and device of the neural network model disclosed in the present invention, a method of determining different quantization parameters according to different data distribution characteristics is adopted. The personalized parameters obtained in this way combine both the local characteristics of the original model and the statistical characteristics of the input data. When processing asymmetric numbers or uneven data, it has better quantization performance than the existing quantization method based on entropy theory.
[0149] The above has been referred to Figures 1 to 8 A quantization method of a neural network model and an apparatus for quantizing a neural network model according to exemplary embodiments of the present disclosure are described.
[0150] Figure 5 , Figure 6 and Figure 8 The various modules in the device of the quantized neural network model of the present disclosure shown can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, each module may correspond to a dedicated integrated circuit, or to a pure software code, or to a module that combines software and hardware. In addition, one or more functions implemented by each module may also be uniformly executed by components in a physical entity device (e.g., a processor, a client or a server, etc.).
[0151] In addition, refer to Figure 2 , Figure 6 and Figure 8 The quantization method of the neural network model of the present disclosure described can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the quantization method of the neural network model of the present disclosure according to the present disclosure.
[0152] The computer program in the computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figure 2 , Figure 6 and / or Figure 8 It is mentioned in the description of the related method, so it will not be repeated here to avoid repetition.
[0153] It should be noted that the various modules in the device of the quantized neural network model according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, each module corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.
[0154] on the other hand, Figure 5 , Figure 6 and Figure 8 The modules shown may also be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor may perform the corresponding operation by reading and running the corresponding program code or code segment.
[0155] For example, the exemplary embodiments of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a computer executable instruction set is stored in the storage component, and when the computer executable instruction set is executed by the processor, a quantization method of a neural network model according to the exemplary embodiments of the present disclosure is executed.
[0156] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing the above instruction set.
[0157] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.
[0158] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0159] According to the exemplary embodiment of the present disclosure, some operations described in the quantization method of the neural network model can be implemented by software, some operations can be implemented by hardware, and furthermore, these operations can also be implemented by a combination of software and hardware.
[0160] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.
[0161] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.
[0162] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.
[0163] The quantization method of the neural network model according to the exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operate according to non-exact boundaries.
[0164] Therefore, refer to Figure 2 , Figure 6 and / or Figure 8 The described quantization method of the neural network model can be implemented by a system including at least one computing device and at least one storage device storing instructions.
[0165] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a method for quantizing a neural network model according to an exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in the storage device. When the computer executable instruction set is executed by the at least one computing device, the execution reference Figure 2 , Figure 6 and / or Figure 8 Quantization method for the described neural network model.
[0166] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.
Claims
1. A quantization method for a neural network model, the quantization method include: Get the neural network model; Calculate the quantization parameter corresponding to the operator to be quantized of the neural network model based on the bisection approximation method; The operator to be quantized of the neural network model is quantized based on the quantization parameter to obtain a neural network model with the quantized operator to be quantized. The step of calculating the quantization parameter corresponding to the operator to be quantized of the neural network model includes: Use the verification data set to verify the neural network model to obtain the input data of each operator to be quantized; Using the binary approximation method, the quantization parameter corresponding to the minimum mean square difference of the input data of each operator to be quantized before and after quantization is calculated according to the input data of each operator to be quantized. The step of calculating the quantization parameter corresponding to the minimum mean square difference comprises: Reduce the dimension of the input data of each operator to be quantized; Based on the statistical characteristics of the input data of each operator to be quantized after dimensionality reduction, the input data of each operator to be quantized after dimensionality reduction is divided into multiple data distribution intervals, and an interval supremum value array is obtained, wherein the interval supremum value array records the supremum value of each data distribution interval; Use the binary approximation method to search for the quantization parameter corresponding to the minimum mean square difference between the starting point and the end point of each data distribution interval. The quantization parameter includes at least one of a truncation parameter of a data distribution interval, a quantization factor parameter, and a truncation factor parameter. Among them, the neural network model is a deep learning neural network model trained to perform one of image recognition and natural language processing.
2. The quantification method according to claim 1, in, The steps for searching the quantization parameters include: For each data distribution interval, when obtaining the interval supremum value array, the minimum average square difference is initialized to the initial average square difference of the data distribution interval, wherein the initial average square difference corresponds to the quantization parameter corresponding to the midpoint between the start point and the end point of the data distribution interval; By bisection approximation between the start point and the end point of the data distribution interval, the average square difference of the approach point of the data distribution interval is calculated, wherein the average square difference of the approach point corresponds to the quantization parameter corresponding to the approach point of the data distribution interval; When the average square difference of the approximation point is less than the minimum average square difference, the minimum average square difference is updated using the average square difference of the approximation point; When the data distribution interval is traversed, the quantization parameter corresponding to the minimum mean square difference is output as the quantization parameter.
3. The quantification method according to claim 1, in, The operators to be quantized of the neural network model include quantizable operators of the neural network model. In which, when the ratio of parameters included in the operator of the neural network model to the total parameters in the neural network exceeds a threshold, or the operator is a computationally intensive operator, the operator is a quantizable operator.
4. The quantization method according to claim 3, wherein the quantization method further comprises: include: Before calculating the quantization parameter corresponding to the quantizable operator, the quantizable operator is marked by inserting a quantization marking operator before the quantizable operator. The step of marking the quantizable operator includes: Determining whether input data of a quantizable operator has weight data; When the input data of the quantizable operator does not have weight data, a quantization marking operator is inserted before the quantizable operator; When the input data of the quantizable operator has weight data, a quantization marking operator is inserted before the quantizable operator, and a quantization marking operator is inserted before the weight data to mark whether the weight data needs to be quantized.
5. A device for quantizing a neural network model, the device include: A data acquisition module configured to acquire a neural network model; A quantization parameter calculation module, configured to calculate a quantization parameter corresponding to an operator to be quantized of a neural network model based on a binary approximation method; A quantization implementation module is configured to quantize the to-be-quantized operator of the neural network model based on the quantization parameter to obtain a neural network model with quantized to-be-quantized operators, The quantization parameter calculation module is configured as follows: Use the verification data set to verify the neural network model to obtain the input data of each operator to be quantized; Using the bisection approximation method, the quantization parameter corresponding to the minimum mean square difference of the input data of each operator to be quantized before and after quantization is calculated from the input data of each operator to be quantized. The quantization parameter calculation module is configured as follows: Reduce the dimension of the input data of each operator to be quantized; Based on the statistical characteristics of the input data of each operator to be quantized after dimensionality reduction, the input data of each operator to be quantized after dimensionality reduction is divided into multiple data distribution intervals, and an interval supremum value array is obtained, wherein the interval supremum value array records the supremum value of each data distribution interval; Use the binary approximation method to search for the quantization parameter corresponding to the minimum mean square difference between the starting point and the end point of each data distribution interval. The quantization parameter includes at least one of a truncation parameter of a data distribution interval, a quantization factor parameter, and a truncation factor parameter. Among them, the neural network model is a deep learning neural network model trained to perform one of image recognition and natural language processing.
6. The device as claimed in claim 5, in, The quantization parameter calculation module is configured as follows: For each data distribution interval, when obtaining the interval supremum value array, the minimum average square difference is initialized to the initial average square difference of the data distribution interval, wherein the initial average square difference corresponds to the quantization parameter corresponding to the midpoint between the start point and the end point of the data distribution interval; By binary approximating the data distribution interval to the middle between the starting point and the end point, the average square difference of the approximation point of the data distribution interval is calculated, wherein the average square difference of the approximation point corresponds to the quantization parameter corresponding to the approximation point of the data distribution interval; When the average square difference of the approximation point is less than the minimum average square difference, the minimum average square difference is updated using the average square difference of the approximation point; When the data distribution interval is traversed, the quantization parameter corresponding to the minimum mean square difference is output as the quantization parameter.
7. The device according to claim 5, in, The operators to be quantized of the neural network model include quantizable operators of the neural network model. Among them, in response to the fact that the proportion of parameters included in the operator of the neural network model to the total parameters in the neural network exceeds a threshold, or it is a computationally intensive operator, the operator is a quantifiable operator.
8. The device according to claim 7, further comprising: include: The quantization marking module is configured to: mark the quantizable operator by inserting a quantization marking operator before the quantizable operator before calculating the quantization parameter corresponding to the quantizable operator, Among them, the quantization marking module is configured as follows: Determining whether input data of a quantizable operator has weight data; When the input data of the quantizable operator does not have weight data, a quantization marking operator is inserted before the quantizable operator; When the input data of the quantizable operator has weight data, a quantization marking operator is inserted before the quantizable operator, and a quantization marking operator is inserted before the weight data to mark whether the weight data needs to be quantized.
9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by one or more computing devices, the one or more computing devices are enabled to implement the quantization method according to any one of claims 1 to 4.
10. A multi-round human-computer dialogue system comprising one or more computing devices and one or more storage devices, wherein the one or more storage devices have a computer program recorded thereon, and when the computer program is executed by the one or more computing devices, the one or more computing devices implement the quantification method as described in any one of claims 1 to 4.
