Method and system for measuring optimal quantization bit width in neural network model
By obtaining the fitting relationship between the data distribution similarity and accuracy loss of the neural network model, and using the fitting relationship and layer weights to allocate the data distribution similarity threshold, the optimal quantization bit width of each layer is determined, which solves the problem of high cost and time overhead in the existing technology and realizes efficient optimal quantization bit width measurement.
Patent Information
- Application Number
- CN202411041587.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2026-02-03
AI Technical Summary
Existing technologies require hardware deployment and traversal of the quantization bit width when determining the optimal quantization bit width for a neural network model, resulting in high deployment costs and time overhead, as well as excessive computation.
By obtaining the fitting relationship between the overall data distribution similarity and accuracy loss before and after model transfer, and using the fitting relationship and the weights of each layer to allocate the data distribution similarity threshold, the optimal quantization bit width of each layer is determined, avoiding traversing the quantization bit width.
Without hardware deployment and traversing the quantization bit width, it reduces deployment costs and time overhead, reduces the number of calculations, and improves the efficiency of finding the optimal quantization bit width.
Smart Images
Figure CN121456492A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of neural network, and particularly relates to a method and system for measuring optimal quantization bit width in a neural network model, a storage medium and a computer device. BACKGROUND
[0002] The existing method for determining the optimal quantization bit width of model migration of a neural network (NN) mainly predicts and evaluates the accuracy loss of the model under different quantization bit widths to find the quantization bit width with the lowest precision that can meet the accuracy loss requirement. There are two main ways to predict the accuracy loss of model migration: one is to directly deploy the model on a new intelligent chip, and the model can directly perform forward inference to obtain the accuracy loss; the other is to simulate through pseudo-quantization technology, and judge the accuracy loss according to the simulation result. Both of them need to traverse the quantization bit width space of each layer to find the optimal quantization bit width that meets the requirement.
[0003] In the existing two methods for predicting the accuracy loss of model migration, the direct deployment method has high deployment cost, including the development and optimization of supporting software such as deep learning high-performance operator library and compiler. The time overhead of the simulation by pseudo-quantization technology is large, and it takes too long to perform a simulation prediction after determining the quantization bit width. At the same time, since each layer of the model has its own quantization bit width, the space of the quantization bit width that can be found is very large, resulting in a sharp increase in the number of times of prediction needed to find the optimal quantization bit width. Combined with the large overhead of each prediction by the above two methods, the overall time and other cost overhead becomes unacceptable.
[0004] The accuracy loss of the quantized model often has an upper limit requirement, which corresponds to an accuracy loss threshold. When the current accuracy loss is equal to the accuracy loss threshold requirement, there is a theoretically lowest-precision optimal quantization bit width. Finding this optimal quantization bit width can reduce the cost of model inference while meeting the upper limit requirement of accuracy loss. Since the model can be quantized in a mixed precision manner, to find the optimal quantization bit width of each layer, it is necessary to traverse various combinations of quantization bit widths, so the time overhead is very large.
[0005] In summary, the prior art is obviously inconvenient and defective in actual use, and needs to be improved. SUMMARY
[0006] In view of the above defects, the purpose of the present application is to provide a method, system, storage medium and computer device for measuring optimal quantization bit width in a neural network model, which can measure the optimal quantization bit width without hardware deployment and traversal quantization bit width, thereby reducing deployment cost, time overhead and calculation times.
[0007] To solve the above technical problems, the present application is implemented as follows:
[0008] In a first aspect, the embodiments of the present application provide a method for measuring optimal quantization bit width in a neural network model, comprising:
[0009] obtaining a fitting relationship between overall data distribution similarity and accuracy loss before and after model migration;
[0010] determining a threshold of overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship;
[0011] allocating the threshold of overall data distribution similarity to each layer of the model according to the weight corresponding to each layer to obtain a threshold of single-layer data distribution similarity of each layer;
[0012] determining optimal quantization bit width of each layer according to the threshold of single-layer data distribution similarity.
[0013] According to the method of the present application, the step of determining optimal quantization bit width of each layer of the model according to the threshold of single-layer data distribution similarity of each layer comprises:
[0014] guiding the model to perform mixed-precision quantization according to the optimal quantization bit width of each layer.
[0015] According to the method of the present application, the step of guiding the model to perform mixed-precision quantization according to the optimal quantization bit width of each layer comprises:
[0016] performing accuracy loss simulation on the quantized model to determine whether the simulated accuracy loss exceeds the accuracy loss threshold.
[0017] According to the method of the present application, the fitting relationship is a fitting curve or fitting relationship formula of the overall data distribution similarity and the accuracy loss;
[0018] The step of determining the threshold of overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship comprises:
[0019] In the fitting curve, searching for the threshold of overall data distribution similarity corresponding to the accuracy loss threshold of the model; or
[0020] According to the accuracy loss threshold of the model and the fitting formula, a threshold corresponding to the overall data distribution similarity is calculated.
[0021] According to the method, the step of determining the optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity comprises:
[0022] According to the threshold of the single-layer data distribution similarity of each layer, the optimal quantization bit width of each layer is determined by gradually reducing the quantization precision; or
[0023] According to the threshold of the single-layer data distribution similarity of each layer, an interval in which the optimal quantization bit width appears is determined, and the optimal quantization bit width of each layer is found by traversing the quantization bit width in the interval.
[0024] According to the method, the step of obtaining the fitting relationship between the overall data distribution similarity and the accuracy loss before and after the model migration comprises:
[0025] The single-layer data distribution similarity of each layer parameter before and after the migration of multiple groups of models with different precisions is calculated.
[0026] According to the weights corresponding to each layer in the model, the single-layer data distribution similarity of each layer is weighted and averaged to obtain multiple groups of the overall data distribution similarity as a measure of accuracy loss.
[0027] The overall data distribution similarity and the accuracy loss of multiple groups are data fitted to obtain the fitting relationship between the two.
[0028] According to the method, the weights corresponding to each layer in the model are the proportion of the calculation amount of each layer in the model to the total calculation amount.
[0029] According to the method, the step of data fitting the overall data distribution similarity and the accuracy loss of multiple groups to obtain the fitting relationship between the two comprises:
[0030] The overall data distribution similarity and the accuracy loss of multiple groups are data fitted to obtain the fitting relationship formula between the two.
[0031]
[0032] In the formula, y is the model accuracy loss, x is the data distribution similarity, and p and q are constants in the formula.
[0033] In a second aspect, an embodiment of the present application provides a measurement system for the optimal quantization bit width of a neural network model, comprising:
[0034] a mathematical fitting module configured to obtain a fitting relationship between an overall data distribution similarity and an accuracy loss before and after model migration;
[0035] a global threshold determination module configured to determine a threshold of the overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship;
[0036] a single-layer threshold allocation module configured to allocate the threshold of the overall data distribution similarity to each layer of the model according to a corresponding weight of each layer to obtain a threshold of a single-layer data distribution similarity of each layer;
[0037] a quantization bit width determination module configured to determine an optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity.
[0038] In a third aspect, an embodiment of the present application provides a storage medium for storing a computer program for executing the measurement method of the optimal quantization bit width of the neural network model.
[0039] In a fourth aspect, an embodiment of the present application provides an electronic device including a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, and the processor implements the measurement method of the optimal quantization bit width of the neural network model when executing the computer program.
[0040] The present application provides a measurement technique of the optimal quantization bit width of the neural network model, obtains a fitting relationship between an overall data distribution similarity and an accuracy loss before and after model migration, determines a threshold of the overall data distribution similarity of the model according to an accuracy loss threshold and the fitting relationship, allocates the threshold of the overall data distribution similarity to each layer according to a corresponding weight of each layer to obtain a threshold of a single-layer data distribution similarity of each layer, and determines an optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity. First, the present application uses the single-layer data distribution similarity in each layer to determine the optimal quantization bit width, which can avoid traversing the quantization bit width of each layer and greatly reduce the time cost. Second, the threshold of the single-layer data distribution similarity of each layer can allocate the consideration of the optimal quantization bit width to each layer instead of the original forward search of the quantization bit width space combined by each layer, which not only reduces the time cost but also greatly reduces the calculation times. Therefore, the present application can measure the optimal quantization bit width without hardware deployment and traversal of the quantization bit width, which not only reduces the deployment cost but also reduces the time cost and the calculation times. BRIEF DESCRIPTION OF DRAWINGS
[0041] Figure 1is a flowchart of a measurement method of an optimal quantization bit width in a neural network model provided by an embodiment one of the present application;
[0042] Figure 2 is a flowchart of a measurement method of an optimal quantization bit width in a neural network model provided by an embodiment two of the present application;
[0043] Figure 3 is a flowchart of a method of predicting accuracy loss of data distribution similarity based on a model provided by an embodiment three;
[0044] Figure 4 is a flowchart of a method of predicting accuracy loss of data distribution similarity based on a model provided by an embodiment four;
[0045] Figure 5 is a structural diagram of a measurement system of an optimal quantization bit width in a neural network model provided by an embodiment one of the present application;
[0046] Figure 6 is a structural diagram of a measurement system of an optimal quantization bit width in a neural network model provided by an embodiment two of the present application;
[0047] Figure 7 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.
[0049] It should be noted that the use of "one embodiment", "an embodiment", "example embodiment", etc. in the specification refers to the described embodiment including a particular feature, structure or characteristic, but not every embodiment must include the particular feature, structure or characteristic. In addition, such expressions do not refer to the same embodiment. Further, when a particular feature, structure or characteristic is described in connection with an embodiment, it is indicated that such a feature, structure or characteristic is incorporated into other embodiments, whether explicitly described or not, within the knowledge of those skilled in the art.
[0050] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.
[0051] The method for measuring the optimal quantization bit width in the neural network model provided by the present invention will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.
[0052] This invention proposes a method for measuring the optimal quantization bit width in a neural network model without hardware deployment or traversing quantization bit widths. This invention is a derivative method based on the model-based method of predicting accuracy loss using data distribution similarity. The basic principle of the model-based method of predicting accuracy loss using data distribution similarity is as follows: calculate the single-layer data distribution similarity of each layer before and after model transfer; weight the single-layer data distribution similarity of each layer according to the proportion of single-layer computation to the total computation to obtain the overall data distribution similarity before and after model transfer; and mathematically fit multiple sets of corresponding overall data distribution similarities and accuracy losses to obtain the fitting relationship between the two.
[0053] This invention obtains a threshold for the overall data distribution similarity based on the fitting relationship between the overall data distribution similarity and accuracy loss before and after model transfer. The threshold for the overall data distribution similarity is then allocated to each layer according to the computational cost. Finally, the optimal quantization bit width of each layer is determined based on the threshold for the single-layer data distribution similarity of each layer.
[0054] Figure 1 This is a flowchart illustrating the method for measuring the optimal quantization bit width in a neural network model provided in Embodiment 1 of the present invention. The method includes the following steps:
[0055] Step S101: Obtain the fitting relationship between the overall data distribution similarity and accuracy loss before and after model transfer.
[0056] The application is a method for predicting accuracy loss of data distribution based on a model, obtaining the fitting relationship between the overall data distribution similarity before and after model migration and the accuracy loss. The purpose is to determine the optimal quantization bit width by calculating the data distribution similarity before and after model migration, thereby solving the problem of high time overhead in finding the optimal quantization bit width of model migration in the prior art.
[0057] In step S102, the threshold of the overall data distribution similarity of the model is determined according to the accuracy loss threshold of the model and the fitting relationship.
[0058] The accuracy loss after model quantization is often required, which corresponds to the accuracy loss threshold. When the current accuracy loss is equal to the accuracy loss threshold requirement, there is a theoretical minimum precision corresponding to the optimal quantization bit width. Finding this optimal quantization bit width can reduce the cost of model inference while meeting the accuracy loss requirement.
[0059] According to the fitting relationship and the requirement of the accuracy loss of the data distribution similarity prediction model, the threshold of the overall data distribution similarity of the model is determined. Compared with the existing pseudo-quantization technology simulation method, the application has smaller time overhead, and can determine the threshold of the overall and each layer data distribution similarity of the model according to the accuracy loss requirement of model migration, thereby effectively reducing the number of calculations.
[0060] In step S103, the threshold of the overall data distribution similarity is allocated to each layer according to the weight corresponding to each layer in the model, to obtain the threshold of the single-layer data distribution similarity of each layer.
[0061] Preferably, the weight corresponding to each layer in the model is the proportion of the calculation amount corresponding to each layer in the model to the total calculation amount. The threshold is allocated to each layer according to the proportion of the calculation amount of each layer to the total calculation amount.
[0062] The application adopts a weighted method according to the proportion of the calculation amount of each layer to the total calculation amount to allocate the threshold of the single-layer data distribution similarity to each layer. This allocation method can effectively allocate the threshold of the single-layer data distribution similarity of each layer under the premise of ensuring the threshold of the overall data distribution similarity of the model, further improving the efficiency of finding the optimal quantization bit width.
[0063] In step S104, the optimal quantization bit width of each layer is determined according to the threshold of the single-layer data distribution similarity.
[0064] In each layer of the model, the optimal quantization bit width of each layer is determined according to the relationship between the threshold of the single-layer data distribution similarity and the quantization bit width, and the threshold of the single-layer data distribution similarity in the layer.
[0065] Preferably, according to the threshold of the single-layer data distribution similarity of each layer, the data before multiplexing is used to infer the optimal quantization bit width of each layer.
[0066] The present application can achieve the effect of determining the optimal quantization bit width of each layer without hardware deployment and traversal quantization bit width, solve the problem of high time overhead of finding the optimal quantization bit width of model migration, and propose a method of determining the optimal quantization bit width through the data distribution similarity threshold of the model.
[0067] Figure 2 is a flowchart of the measuring method of the optimal quantization bit width of the neural network model provided in Embodiment Two of the present application, and the method comprises the following steps:
[0068] Step S201: Obtain the fitting relationship between the overall data distribution similarity and the accuracy loss before and after model migration.
[0069] Step S202: According to the accuracy loss threshold of the model and the fitting relationship, determine the threshold of the overall data distribution similarity of the model.
[0070] Preferably, the fitting relationship is a fitting curve or a fitting relationship formula of the overall data distribution similarity and the accuracy loss.
[0071] The step S202 can further comprise:
[0072] In the fitting curve, search for the threshold of the overall data distribution similarity corresponding to the accuracy loss threshold of the model; or
[0073] According to the accuracy loss threshold of the model and the fitting relationship formula, calculate the threshold of the overall data distribution similarity corresponding to the accuracy loss threshold of the model.
[0074] Step S203: According to the corresponding weights of each layer in the model, distribute the threshold of the overall data distribution similarity to each layer to obtain the threshold of the single-layer data distribution similarity of each layer.
[0075] Step S204: According to the threshold of the single-layer data distribution similarity, determine the optimal quantization bit width of each layer.
[0076] Preferably, the step S204 further comprises:
[0077] (1) According to the threshold of the single-layer data distribution similarity of each layer, use gradually reduced quantization precision to determine the optimal quantization bit width of each layer. Since the parameter precision of each layer and the threshold of the single-layer data distribution similarity of the layer have the same trend, the quantization precision can be gradually reduced to find the lowest precision quantization bit width that meets the requirements; or
[0078] (2) According to the threshold of the single-layer data distribution similarity of each layer, the interval in which the optimal quantization bit width appears is determined, and the quantization bit width is traversed in the interval to find the optimal quantization bit width of each layer. The optimal quantization bit width of each layer is calculated according to the threshold of the single-layer data distribution similarity of each layer of the model, which can also be directly reused from the data used in curve fitting. Since the single-layer data distribution similarity in the layer has been calculated at many different quantization bit width nodes during curve fitting, the previous data can be viewed to determine an interval in which the optimal quantization bit width appears, and then the quantization bit width is traversed in the interval to find the lowest precision quantization bit width.
[0079] Step S205, according to the optimal quantization bit width of each layer, guiding the model to perform mixed-precision quantization.
[0080] Preferably, the mixed-precision quantization is guided by the configuration file of the quantization bit width threshold of each layer.
[0081] Step S206, simulating the accuracy loss of the quantized model, and judging whether the simulated accuracy loss exceeds the accuracy loss threshold.
[0082] Preferably, a simulator (also known as a simulator) can be used to test whether the inference loss after quantization is within the preset inference loss threshold.
[0083] The measurement method of the optimal quantization bit width of the neural network model provided in the second embodiment has the following three technical key points:
[0084] Key point 1, according to the threshold of the single-layer data distribution similarity of each layer, the optimal quantization bit width of the layer is determined. In the method of predicting accuracy loss based on the data distribution similarity of the model, the relationship between the quantization bit width of each layer of the model and the single-layer data distribution similarity of the layer is known. The lower the precision of the quantization loss, the higher the data distribution similarity, that is, the threshold of the single-layer data distribution similarity of each layer will correspond to the optimal quantization bit width of the layer. The method of determining the optimal quantization bit width by using the single-layer data distribution similarity in the layer can avoid traversing the quantization bit width of each layer, greatly reducing the time cost.
[0085] Key point 2, use the fitting relationship, fitting curve or fitting formula, determine the threshold of data distribution similarity according to the accuracy loss requirement of model migration. The time overhead of calculating the data distribution similarity is less than the time overhead of the pseudo-quantization technique simulation, and at the same time, this way can be used to determine the threshold of the data distribution similarity of the model as a whole and each layer from the accuracy loss requirement of model migration, and the consideration of the optimal quantization bit width can be allocated to each layer according to the threshold of the single-layer data distribution similarity of each layer, instead of the original forward to find the quantization bit width space of the combination of each layer. That is, the way of calculating the model data distribution similarity not only has less time overhead in prediction, but also greatly reduces the calculation times.
[0086] Key point 3, a weighted way according to the proportion of the calculation amount of each layer in the total calculation amount is used to allocate the threshold of the single-layer data distribution similarity of each layer. According to the accuracy loss requirement of model migration, the threshold of the overall data distribution similarity of the model can be obtained, and the threshold of the single-layer data distribution similarity of each layer of the model needs to be allocated in a reasonable way.
[0087] In the process of the method for predicting the accuracy loss based on the data distribution similarity of the model, through observation and analysis of the parameters before and after model migration, two observation conclusions are finally obtained: the data distribution of each layer parameter of the deep learning model is regular, and is approximately normally distributed. The data distribution of the model parameters will change to different degrees due to the different hardware characteristics after the model is migrated to different hardware, and the lower the similarity of the data distribution of the parameters before and after the model migration, the higher the accuracy loss. According to the two observation conclusions, the application conceives a method for measuring the accuracy loss of model migration, specifically a pre-deployment estimation method, that is, simulating the parameters after model migration, and statistically obtaining the data distribution similarity of each layer parameter before and after model migration, and obtaining the pre-estimation of the overall accuracy loss of the model through weighted average of the data distribution similarity of each layer.
[0088] Figure 3 is the flowchart of the method for predicting the accuracy loss based on the data distribution similarity of the model provided by the third embodiment of the application, and the steps S101 shown in the figure Figure 1 The method comprises the following steps:
[0089] Step S301, calculate the single-layer data distribution similarity of each layer parameter before and after model migration of multiple groups of models with different precisions.
[0090] Preferably, the JS (Jensen Shannon) divergence of each layer parameter before and after model migration of different precision is calculated, and the JS divergence is taken as the single layer data distribution similarity.
[0091] In step S302, the single layer data distribution similarity of each layer is weighted and averaged according to the weight corresponding to each layer in the model, and the overall data distribution similarity of multiple groups is obtained as the measurement of accuracy loss.
[0092] Preferably, the weight corresponding to each layer in the model is the proportion of the calculation amount of each layer in the total calculation amount.
[0093] In step S303, the overall data distribution similarity of multiple groups and the accuracy loss are data fitted to obtain the fitting relationship between them.
[0094] The present application needs to be carried out on the result of the data distribution similarity prediction model accuracy loss method, and the fitting relationship formula of the accuracy loss and the overall data distribution similarity threshold of the model can be obtained. According to the given accuracy loss threshold, the corresponding overall data distribution similarity threshold can be obtained in the fitting relationship. After obtaining the accuracy loss threshold, the corresponding overall data distribution similarity threshold is found in the fitting curve, or the overall data distribution similarity threshold is calculated according to the fitting relationship formula.
[0095] Preferably, the first overall data distribution similarity of multiple groups and the first accuracy loss are data fitted to obtain the fitting relationship between them, and the fitting relationship formula of the accuracy loss y and the overall data distribution similarity x has the following form:
[0096]
[0097] Wherein, y in the formula is the model accuracy loss, x is the data distribution similarity, and p and q in the formula are constants. The fitting relationship formula is to curve fit the model accuracy loss and the data distribution similarity, and is a formula form that is found to be more suitable for the relationship between them.
[0098] After obtaining the acceptable accuracy loss y, a complex fractional equation needs to be solved to obtain the value of the overall data distribution similarity x, so the corresponding point can be directly found in the curve to obtain the threshold of the overall data distribution similarity. Of course, the fractional equation can also be solved by computer, such as calling the fslove function in python.
[0099] After obtaining the threshold of the overall data distribution similarity, the threshold of the overall data distribution similarity is distributed to each layer of the model by the weighted distribution method according to the proportion of the calculation amount of each layer in the total calculation amount, and the threshold of the single layer data distribution similarity of each layer of the model is obtained.
[0100] Preferably, the step S303 can further include:
[0101] According to the fitting formula, the accuracy loss corresponding to the overall data distribution similarity of the current precision is calculated.
[0102] In order to solve the problem of high cost of measuring the accuracy loss of model migration of neural network in actual deployment, the present application provides a model migration accuracy loss measurement method, which uses the weighted average of the data distribution similarity of the inter-layer parameters before and after model migration to estimate the accuracy loss in advance. The present application uses the data distribution similarity as the simulation prediction method for measuring the accuracy loss, which does not require high deployment cost compared with the existing direct deployment scheme. At the same time, the present application can accurately and safely reflect the accuracy loss of model migration under various precisions, so as to help the model user to find the appropriate migration precision, select a more accurate migration model and intelligent chip, and ensure the performance and safety of the model. Therefore, the present application can more accurately and safely predict the accuracy loss of model migration, and has lower cost.
[0103] Figure 4 is the flowchart of the method for predicting the accuracy loss based on the data distribution similarity of the model according to the fourth embodiment of the present application, which is further refined for the step S101 of obtaining the fitting relationship between the overall data distribution similarity and the accuracy loss before and after model migration as shown in Figure 1 The method includes the following steps:
[0104] Step S401: training the full-precision model to obtain the parameters of each layer before model migration.
[0105] Step S402: simulating the parameters of each layer after model migration under different precisions.
[0106] Preferably, the model is pseudo-quantized under different precisions to obtain the parameters of each layer after model migration after pseudo-quantization.
[0107] The parameters of each layer after model migration after pseudo-quantization are dequantized to obtain the corresponding parameters of each layer after model migration after dequantization. Preferably, the dequantization formula is:
[0108] r=S(q-Z)
[0109]
[0110] wherein r represents a floating-point real number, q represents a quantized fixed-point integer, q max , q min are the maximum and minimum values of q respectively, r max , r minSimilarly, S represents the proportional relationship between floating-point numbers and fixed-point integers, and Z represents the integer corresponding to the quantization of 0 in floating-point real numbers.
[0111] In step S403, the single-layer data distribution similarity of the multiple groups of parameters of each layer before and after model migration under different precisions is calculated according to the parameters of each layer before model migration and the parameters of each layer after model migration.
[0112] Preferably, the JS divergence of the multiple groups of parameters of each layer before and after model migration under different precisions is calculated, and the JS divergence is taken as the single-layer data distribution similarity.
[0113] Preferably, the single-layer data distribution similarity of the multiple groups of parameters of each layer before and after model migration under different precisions is calculated according to the parameters of each layer before model migration and the parameters of each layer before model migration after dequantization.
[0114] Specifically, to obtain the data distribution similarity before and after model migration, the parameters of each layer of the pre-migration model and the post-migration model are needed. The pre-migration model is a full-precision model. Since the model cannot be actually deployed, the parameters of each layer of the post-migration model are obtained by a pseudo-quantization algorithm. After obtaining the parameters of each layer of the pre-migration model after dequantization, the data distribution similarity of the corresponding model parameters needs to be calculated. In order to make the parameters of the pseudo-quantization model and the parameters of the full-precision model in the same range, the parameters of each layer of the pre-migration model after dequantization need to be dequantized.
[0115] After obtaining the parameters of each layer of the pre-migration model after dequantization, the data distribution similarity of the parameters is calculated, preferably the JS divergence of the two is calculated.
[0116] In step S404, the single-layer data distribution similarity of each layer is weighted and averaged to obtain multiple groups of overall data distribution similarities as the measure of accuracy loss.
[0117] Preferably, the weight corresponding to each layer in the model is the proportion of the calculation amount of each layer in the total calculation amount. After obtaining the single-layer data distribution similarity of each layer, the proportion of the calculation amount of each layer in the total calculation amount is taken as the weight, the single-layer data distribution similarity of each layer is weighted and summed to obtain the overall data distribution similarity, which is finally taken as the measure of accuracy loss before and after model migration, and then the model accuracy loss needs to be calculated under the precision after pseudo-quantization.
[0118] In step S405, the multiple groups of overall data distribution similarities and the accuracy loss are data fitted to obtain the fitting relationship between the two.
[0119] Preferably, the fitting relationship between the multiple groups of overall data distribution similarities and the accuracy loss is:
[0120]
[0121] wherein y is the model accuracy loss, x is the data distribution similarity, and p and q are constants in the formula. The fitting formula is used to curve fit the model accuracy loss and the data distribution similarity, and is a formula form that is found to be more suitable for the relationship between the two.
[0122] Since multiple sets of data are needed to fit the model data distribution similarity and the model accuracy loss, the above operations of calculating the model data distribution similarity and determining the corresponding model accuracy loss need to be repeated at multiple different pseudo-quantization nodes to obtain multiple sets of relationship data. Then the numerical values of the model data distribution similarity and the model accuracy loss need to be mathematically fitted.
[0123] After the fitted rational function model and the multiple sets of corresponding data are determined, the Matlab fit function can be used for mathematical fitting to obtain the fitting formula of the model data distribution similarity and the model accuracy loss.
[0124] Step S406, calculate the single-layer data distribution similarity of each layer of the model before and after migration of multiple sets of model at the current precision.
[0125] Step S407, according to the weights corresponding to each layer in the model, the single-layer data distribution similarity of each layer is weighted and averaged to obtain multiple sets of overall data distribution similarity.
[0126] Step S408, according to the fitting formula, calculate the accuracy loss corresponding to the overall data distribution similarity at the current precision.
[0127] After obtaining the fitting formula, the accuracy loss of other quantization nodes of the model can be predicted. The method is to calculate the overall data distribution similarity of the model in the manner described above, and then find the corresponding model accuracy loss value in the fitting image after obtaining the data distribution similarity value, or calculate the model accuracy loss value according to the fitting formula.
[0128] The technical scheme of embodiment two provided by the present application includes the following key points:
[0129] Key point 1: The data distribution similarity before and after model migration is used as the measure of accuracy loss, and the JS divergence is used as the measure of data distribution similarity. When selecting the measure of data distribution similarity, it is finally determined to use the JS (Jensen Shannon) divergence derived from the KL (Kullback Leibler) divergence as the measure of data distribution similarity, which has the following advantages:
[0130] 1. Symmetry: JS divergence can measure the similarity between two data distributions simultaneously, regardless of their order.
[0131] 2. Normalization: The value range of JS divergence is between 0 and 1, which makes its results easy to interpret and compare. The closer the value is to 0, the higher the similarity of the two distributions.
[0132] 3. Smoothness: JS divergence is calculated by normalizing the average of KL divergence, which helps to reduce errors caused by noise or uncertainty in data distribution.
[0133] The data distribution similarity before and after model migration can effectively measure the accuracy loss of single layer after model migration. Using JS divergence as the measure of data distribution similarity also has the advantages of symmetry, normalization, and smoothness.
[0134] Key point 2: For the overall data distribution similarity of the model, the invention proposes a weighted average method for single layer data distribution similarity based on the importance of each layer of the model, to obtain the overall data distribution similarity before and after model migration. At the same time, the proportion of the calculation amount in the layer to the total calculation amount is used as the weight to measure the importance of each layer, rather than selecting the parameter amount of each layer as the measure.
[0135] The overall data distribution similarity before and after model migration obtained by the weighted average method with calculation amount as weight has good effect on measuring the accuracy loss of model migration. The tool in the invention has good prediction performance in AlexNet and ResNet models with cifar100 as the data set.
[0136] Key point 3: In the process of fitting data distribution similarity-accuracy loss, it is necessary to determine the fitting relationship with good effect. Among the enumeration of various common relationship formulas, the invention selects the best fitting relationship formula:
[0137]
[0138] The fitting effect of the above fitting relationship formula is the best. After fitting the AlexNet and VGG_19 models with cifar100 as the data set, the R2 values are 0.9732 and 0.9818 respectively, which is very close to 1, and the fitting effect is good.
[0139] In this embodiment four, the model is pseudo-quantized at different precisions, the dequantization values of the parameters in each layer of the model after pseudo-quantization are calculated, and then the data distribution similarity with the full-precision model parameters is calculated. After obtaining the single-layer data distribution similarity of each layer, the single-layer data distribution similarity of each layer of the model is weighted and summed to obtain the overall data distribution similarity according to the weight of the proportion of the calculation amount of each layer, and then the accuracy loss of the model at the current precision is calculated. The mathematical relationship between the overall data distribution similarity and the model accuracy loss is obtained by mathematical fitting. When predicting the model accuracy loss, only the overall data distribution similarity calculated is brought into the mathematical relationship to calculate the corresponding model accuracy loss.
[0140] The measurement method of the optimal quantization bit width in the neural network model provided by the embodiment of the present application has the following technical effects: first, the method of determining the optimal quantization bit width by using the single-layer data distribution similarity in the layer can avoid traversing the quantization bit width of each layer, greatly reducing the time cost. Second, according to the threshold of the single-layer data distribution similarity of each layer, the consideration of the optimal quantization bit width can be allocated to each layer, rather than the original forward search for the combined quantization bit width of each layer. This not only reduces the time cost, but also greatly reduces the number of calculations. Therefore, the present application can measure the optimal quantization bit width without hardware deployment and traversal of the quantization bit width, which not only reduces the deployment cost, but also reduces the time cost and the number of calculations.
[0141] It should be noted that the measurement method of the optimal quantization bit width in the neural network model provided by the embodiment of the present application can be an electronic device, a measurement system of the optimal quantization bit width in the neural network model, or a control module in the measurement system of the optimal quantization bit width in the neural network model for executing the measurement method of the optimal quantization bit width in the neural network model. In the embodiment of the present application, the measurement system of the optimal quantization bit width in the neural network model is taken as an example to illustrate the measurement system of the optimal quantization bit width in the neural network model provided by the embodiment of the present application.
[0142] Figure 5 FIG. 1 is a structural schematic diagram of the measurement system of the optimal quantization bit width in the neural network model provided by the first embodiment of the present application, which at least includes a mathematical fitting module 10, an overall threshold determination module 20, a single-layer threshold distribution module 30, and a quantization bit width determination module 40, wherein:
[0143] The mathematical fitting module 10 is configured to obtain the fitting relationship between the overall data distribution similarity and the accuracy loss before and after model migration.
[0144] The overall threshold determination module 20 is configured to determine a threshold of the overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship.
[0145] The single-layer threshold allocation module 30 is configured to allocate the threshold of the overall data distribution similarity to each layer according to the weight corresponding to each layer in the model, to obtain a single-layer data distribution similarity threshold of each layer.
[0146] The quantization bit width determination module 40 is configured to determine the optimal quantization bit width of each layer according to the single-layer data distribution similarity threshold.
[0147] Figure 6 is a structural schematic diagram of a neural network model optimal quantization bit width measurement system provided in Embodiment Two of the present application. The neural network model optimal quantization bit width measurement system 100 at least includes a mathematical fitting module 10, an overall threshold determination module 20, a single-layer threshold allocation module 30, a quantization bit width determination module 40, a quantization processing module 50, and a loss simulation module 60, wherein:
[0148] The mathematical fitting module 10 is configured to obtain a fitting relationship between the overall data distribution similarity and the accuracy loss before and after model migration. Preferably, the fitting relationship is a fitting curve or a fitting relationship formula of the overall data distribution similarity and the accuracy loss. The mathematical fitting module 10 further includes a calculation submodule 11, a weighting submodule 12, and a fitting submodule 13, wherein:
[0149] The calculation submodule 11 is configured to calculate the single-layer data distribution similarity of each layer parameter of multiple groups of models before and after migration at different precisions.
[0150] The weighting submodule 12 is configured to perform weighted average calculation on the single-layer data distribution similarity of each layer according to the weight corresponding to each layer in the model, to obtain multiple groups of overall data distribution similarities as a measure of accuracy loss. Preferably, the weight is the proportion of the calculation amount of each layer in the model to the total calculation amount.
[0151] The fitting submodule 13 is configured to perform data fitting on the multiple groups of overall data distribution similarities and the accuracy loss, to obtain a fitting relationship therebetween. Preferably, the fitting relationship formula of the multiple groups of overall data distribution similarities and the accuracy loss is obtained by data fitting.
[0152]
[0153] The fitting relationship formula is used to perform curve fitting on the model accuracy loss and the data distribution similarity, and is a formula form that is found to be more suitable for the relationship between the two.
[0154] The whole threshold determination module 20 is configured to determine the threshold of the whole data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship. Preferably, the whole threshold determination module 20 further comprises a first threshold determination submodule 21 or a second threshold determination submodule 22, wherein:
[0155] The first threshold determination submodule 21 is configured to search for the threshold of the whole data distribution similarity corresponding to the accuracy loss threshold of the model in the fitting curve.
[0156] The second threshold determination submodule 22 is configured to calculate the threshold of the whole data distribution similarity corresponding to the accuracy loss threshold of the model according to the fitting relationship.
[0157] The single-layer threshold allocation module 30 is configured to allocate the threshold of the whole data distribution similarity to each layer of the model according to the corresponding weight of each layer to obtain the threshold of the single-layer data distribution similarity of each layer.
[0158] The quantization bit width determination module 40 is configured to determine the optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity. Preferably, the quantization bit width determination module 40 further comprises a first quantization bit width submodule 41 or a second quantization bit width submodule 42, wherein:
[0159] The first quantization bit width submodule 41 is configured to determine the optimal quantization bit width of each layer by gradually reducing the quantization precision according to the threshold of the single-layer data distribution similarity of each layer.
[0160] The second quantization bit width submodule 42 is configured to determine the interval in which the optimal quantization bit width appears according to the threshold of the single-layer data distribution similarity of each layer, and traverse the quantization bit width in the interval to find the optimal quantization bit width of each layer.
[0161] The quantization processing module 50 is configured to guide the model to perform mixed-precision quantization according to the optimal quantization bit width of each layer.
[0162] The loss simulation module 60 is configured to simulate the accuracy loss of the quantized model, and determine whether the simulated accuracy loss exceeds the accuracy loss threshold.
[0163] The measurement system for the optimal quantization bit width in the neural network model provided by the embodiment of the present application can realize the following advantages: Figures 1 to 4 The measurement method for the optimal quantization bit width in the neural network model is shown in the embodiment, and each process is implemented. To avoid repetition, it will not be described here.
[0164] The measurement system for the optimal quantization bit width in the neural network model provided by the embodiment of the present application determines the optimal quantization bit width by calculating the data distribution similarity before and after model migration, thereby solving the problem of high time overhead in the prior art for finding the optimal quantization bit width for model migration. Compared with the existing pseudo-quantization technology simulation method, the present application has smaller time overhead, and can reversely determine the threshold of the data distribution similarity of the model as a whole and each layer according to the requirement of the accuracy loss of model migration, thereby effectively reducing the number of calculations. In addition, the present application adopts a weighted method according to the proportion of the calculation amount of each layer in the total calculation amount to allocate the threshold of the single-layer data distribution similarity to each layer. This allocation method can effectively allocate the threshold of the single-layer data distribution similarity to each layer under the premise of ensuring the threshold of the overall data distribution similarity of the model, and further improves the efficiency of finding the optimal quantization bit width.
[0165] The present application also provides a storage medium for storing the measurement method for the optimal quantization bit width in the neural network model. Figures 1 to 4 The computer program, for example, computer program instructions, when executed by a computer, can call or provide the method and / or technical solutions according to the present application through the operation of the computer, and can achieve the same technical effects. To avoid repetition, details are not repeated here. The program instructions for calling the method of the present application can be stored in a fixed or removable storage medium, and / or transmitted and / or stored in the storage medium of the computer device running the program instructions.
[0166] According to one embodiment of the present application, the present application also provides an electronic device 400 as shown in the accompanying drawings. Figure 7 The electronic device 400 can optionally include a storage medium 200 for storing a computer program and a processor 300 for executing the computer program, wherein the computer program is executed by the processor 300 to implement the measurement method for the optimal quantization bit width in the neural network model, triggers the electronic device 400 to execute the method and / or technical solutions based on the foregoing embodiments, and can achieve the same technical effects. To avoid repetition, details are not repeated here. It should be noted that the electronic device in the embodiment of the present application includes mobile electronic devices and non-mobile electronic devices. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer, a netbook or a personal digital assistant, etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not limited specifically.
[0167] It should be noted that the present application can be implemented in software and / or in a combination of software and hardware, e.g., using application specific integrated circuits (ASIC), a general purpose computer or any other similar hardware devices. In one embodiment, the software program of the present application is implemented by the processor to perform the above steps or functions. Also, the software program of the present application (including related data structures) can be stored on a computer readable medium, e.g., a RAM memory, a magnetic or optical drive or diskette, and the like. Additionally, some of the steps or functions could be implemented in hardware, e.g., as circuitry which co-operates with the processor in performing the various steps or functions.
[0168] The present application can be implemented on a computer as a computer implemented method, or in dedicated hardware, or in a combination of both. Executable code for a method according to the present application, or parts thereof, can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software etc. Alternatively, the computer program product comprises non-transitory program code means stored on a computer readable medium to perform a method according to the present application when that program product is executed on a computer.
[0169] In an alternative embodiment, the computer program comprises computer program code means adapted to perform all the steps of a method according to the present application when the computer program is run on a computer. Alternatively, the computer program is embodied on a computer readable medium.
[0170] It should be noted that, as used in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the steps of the methods and the order of the steps in the methods of the embodiments of the present application are not limited to the order presented in the figures or discussed herein. Further, the features of the various examples described herein can be combined with each other, as appropriate.
[0171] Of course, the present application can have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, and these corresponding changes and modifications shall all belong to the protection scope of the claims of the present application.
Claims
1. A method for measuring the optimal quantization bit width in a neural network model, characterized in that, The method comprises the following steps: Obtaining the fitting relationship between the overall data distribution similarity and the accuracy loss of the model before and after migration; According to the accuracy loss threshold of the model and the fitting relationship, determining the threshold of the overall data distribution similarity of the model; According to the weights corresponding to each layer of the model, assigning the threshold of the overall data distribution similarity to each layer to obtain the threshold of the single-layer data distribution similarity of each layer; According to the threshold of the single-layer data distribution similarity, determining the optimal quantization bit width of each layer.
2. The method of claim 1, wherein the neural network model is a quantized neural network model. After the step of determining the optimal quantization bit width of each layer in the model according to the threshold of the single-layer data distribution similarity, the method further comprises the following steps: According to the optimal quantization bit width of each layer, guiding the model to perform mixed-precision quantization.
3. The method of claim 2, wherein the method further comprises: After the step of guiding the model to perform mixed-precision quantization according to the optimal quantization bit width of each layer, the method further comprises the following steps: Simulating the accuracy loss of the quantized model to determine whether the simulated accuracy loss exceeds the accuracy loss threshold.
4. The method of claim 1, wherein the neural network model is a convolutional neural network (CNN) model. The fitting relationship is a fitting curve or a fitting formula of the overall data distribution similarity and the accuracy loss; The step of determining the threshold of the overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship comprises the following steps: In the fitting curve, searching for the threshold of the overall data distribution similarity corresponding to the accuracy loss threshold of the model; or According to the accuracy loss threshold of the model and the fitting formula, calculating the corresponding threshold of the overall data distribution similarity. The step of determining the optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity comprises the following steps:
5. The method of claim 1, wherein the neural network model is a convolutional neural network (CNN) model. According to the threshold of the single-layer data distribution similarity of each layer, determining the optimal quantization bit width of each layer by gradually reducing the quantization precision; or According to the threshold of the single-layer data distribution similarity of each layer, determining the interval where the optimal quantization bit width appears, and traversing the quantization bit width in the interval to find the optimal quantization bit width of each layer. The step of obtaining the fitting relationship between the overall data distribution similarity and the accuracy loss of the model before and after migration comprises the following steps:
6. The method of claim 1, wherein the neural network model is a convolutional neural network (CNN) model. Calculating the single-layer data distribution similarity of each layer parameter before and after migration of multiple groups of models with different precisions; According to the weights corresponding to each layer of the model, performing weighted average calculation on the single-layer data distribution similarity of each layer to obtain multiple groups of the overall data distribution similarity as the measurement of accuracy loss; Data fitting is performed on multiple groups of the overall data distribution similarity and the accuracy loss to obtain the fitting relationship between the two. The weights corresponding to each layer of the model are the proportion of the calculation amount of each layer in the total calculation amount.
7. The method of claim 6, wherein the method further comprises: The step of data fitting multiple groups of the overall data distribution similarity and the accuracy loss to obtain the fitting relationship between the two comprises the following steps:
8. The method of claim 6, wherein the method further comprises: The fitting formula of the overall data distribution similarity and the accuracy loss is: Wherein, y is the model accuracy loss, x is the data distribution similarity, and p and q are constants in the formula.
9. A measurement system for measuring the optimal quantization bit-width of a neural network model constructed based on the method of any one of claims 1-8. The method comprises the following steps: a mathematical fitting module, configured to obtain a fitting relationship between the overall data distribution similarity and the accuracy loss of the model before and after model migration; an overall threshold determination module, configured to determine a threshold of the overall data distribution similarity of the model according to the accuracy loss threshold of the model and the fitting relationship; a single-layer threshold allocation module, configured to allocate the threshold of the overall data distribution similarity to each layer of the model according to the corresponding weight of each layer, to obtain a threshold of the single-layer data distribution similarity of each layer; a quantization bit width determination module, configured to determine the optimal quantization bit width of each layer according to the threshold of the single-layer data distribution similarity.
10. A storage medium, characterized by A computer program for storing a measurement method of the optimal quantization bit width in the neural network model according to any one of claims 1-8.
11. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, The processor executes the computer program to realize the measurement method of the optimal quantization bit width in the neural network model according to any one of claims 1-8.