Quantization method and quantization device for deep learning model

Abstract

A quantization method and a quantization device for a deep learning model are disclosed, including: looping the following steps until a preset condition is met: quantizing a first model based on a quantization parameter to obtain a second model; testing the second model to obtain actual values ​​of multiple optimization target parameters; calculating a loss function based on the actual values ​​of multiple optimization target parameters and the expected values ​​of multiple optimization target parameters and the constraint values ​​of multiple optimization target parameters; updating the quantization parameters based on the calculated loss function, and using the second model as the first model; in response to the looped operation steps satisfying the preset condition, obtaining the optimal quantization parameters, and using the first model quantized based on the optimal quantization parameters as the final quantized model. The model obtained by quantization using the present disclosure has a faster convergence speed, a good compression effect of the model, and a mixture of high-precision quantization and low-precision quantization, which can ensure the accuracy of the model without fine-tuning.

Description

Technical Field

[0001] The present disclosure relates to quantization of deep learning models, and more particularly, to a quantization method and a quantization device for deep learning models. Background Art

[0002] With the development of artificial intelligence (AI) technology, it can be applied in many fields. Among them, deep learning, as an important branch of AI, has made breakthroughs in computer vision, language processing, and text processing. At present, deep learning technology is applied in both mobile and data center services. However, at this stage, deep learning technology has high requirements on the computing power of related hardware devices, large memory usage, and high power consumption in the application stage, which results in heavy load on mobile applications or data center services.

[0003] In the related technology, quantized deep learning model (for example, neural network model) technology is used to utilize low-precision, low-power neural network chips (for example, neural network processing unit (NPU), tensor processing unit (TPU), field programmable gate array (FPGA), etc.) to reduce energy consumption and reduce inference latency, so as to be suitable for mobile applications or data center services.

[0004] However, the quantized deep learning model technical solutions in related technologies have at least the following disadvantages: (1) the quantized model training of the training-perceptual quantization technology converges slowly; (2) the model is quantized to a fixed precision, resulting in limited compression effect of the quantized model; (3) low-precision quantization results in a large loss of accuracy, and fine-tuning is required to restore accuracy. Summary of the invention

[0005] The present disclosure provides a quantization method and a quantization device for a deep learning model to at least solve the problems in the above-mentioned related technologies, or may not solve any of the above-mentioned problems.

[0006] According to one aspect of an exemplary embodiment of the present disclosure, a quantization method for a deep learning model is provided, comprising: looping the following operation steps until a preset condition is met: quantizing a first model based on a quantization parameter to obtain a second model; testing the second model to obtain actual values ​​of multiple optimization target parameters; calculating a loss function based on the actual values ​​of the multiple optimization target parameters and the expected values ​​of the multiple optimization target parameters and the constraint values ​​of the multiple optimization target parameters; updating the quantization parameters based on the calculated loss function, and using the second model as the first model; in response to the looped operation steps satisfying the preset condition, obtaining the optimal quantization parameter, and using the first model quantized based on the optimal quantization parameter as the final quantized model.

[0007] Through this embodiment, quantization is performed based on quantization parameters, so that quantization can be performed for different precision types of quantization parameters, avoiding the problem of quantizing the model to a fixed precision and poor compression effect, and can also ensure the mixing of high-precision quantization and low-precision quantization; furthermore, the actual values, expected values, and constraint values ​​of multiple optimization target parameters are integrated to obtain the loss function in the optimization process, so that the conflict relationship between the optimization targets can be fully considered and unified optimization can be performed; in addition, the present disclosure directly quantizes the pre-trained model and does not complete the quantization in the training stage, avoiding the problem of slow convergence of quantization model training when quantization is completed in the training stage. Therefore, compared with the quantized model of the prior art, the quantized model of the present disclosure has a faster convergence speed, good compression effect of the model, and a mixture of high-precision quantization and low-precision quantization, which can ensure the accuracy of the model without fine-tuning.

[0008] Optionally, the quantization of the first model based on the quantization parameters to obtain the second model includes: performing quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; performing quantization configuration on the simulation quantization model based on the quantization parameters; calculating quantization coefficients based on the simulation quantization model after quantization configuration; and performing model rewriting on the simulation quantization model based on the quantization coefficients to obtain the second model. Through this embodiment, each layer of the model can be configured with appropriate quantization parameters, so that the corresponding precision type can be used according to the quantization parameters, ensuring the mixing of high-precision quantization and low-precision quantization, and comprehensively utilizing the advantages of each precision quantization to achieve a better quantization effect.

[0009] Optionally, performing quantization labeling on each operator to be quantized in the first model includes: inserting a simulation quantization operator into each operator to be quantized, wherein the simulation quantization operator includes a simulation quantization operator for quantization weights and a simulation quantization operator for quantization activation values. Through this embodiment, quantization labeling can be performed conveniently and quickly.

[0010] Optionally, based on the quantization parameters, the simulation quantization model is configured for quantization, including: analyzing the correspondence between the quantization parameters and the hierarchical order of operators in the simulation quantization model; and configuring the quantization parameters to the corresponding operators according to the correspondence. Through this embodiment, according to the correspondence between the quantization parameters and the hierarchical order of operators in the simulation quantization model, the configuration of high-precision quantization and low-precision quantization is completed, thereby preventing confusion of the quantization parameters of each operator in the subsequent optimization process.

[0011] Optionally, based on the simulation quantization model after quantization configuration, the quantization coefficient is calculated, including: determining the precision type of the data corresponding to the quantization parameter configured by each simulation quantization operator inserted in the simulation quantization model; simulating the truncation error and rounding error from floating point numbers to integers based on the mapping relationship between floating point data and the data of the precision type, and then calculating the quantization coefficient of each simulation quantization operator. The quantization coefficient obtained by this embodiment can make the model quantized better.

[0012] Optionally, the simulation quantization model is rewritten based on the quantization coefficient to obtain the second model, including: for each simulation quantization operator in the simulation quantization model, determining the quantization parameter and the corresponding quantization coefficient of the simulation quantization operator; and replacing the simulation quantization operator with a low-precision operator that supports the quantization parameter and the quantization coefficient. Through this embodiment, the quantization of the model is achieved.

[0013] Optionally, the target parameter includes accuracy and at least one of the following parameters: size of the quantization model, energy consumption, and inference latency. Through this embodiment, some target parameters that can be optimized are given.

[0014] Optionally, based on the actual values ​​of multiple optimization target parameters and the expected values ​​of multiple optimization target parameters and the constraint values ​​of multiple optimization target parameters, the loss function is calculated, including: based on the difference between the expected values ​​of the multiple optimization target parameters and the corresponding actual values ​​and the difference between the constraint values ​​of the multiple optimization target parameters and the corresponding actual values, the loss function is calculated. Through this embodiment, the loss function is calculated according to the difference between the expected value, the constraint value and the actual value, so that the loss function can be easily obtained and the complexity of the calculation can be reduced.

[0015] Optionally, the function expression of the loss function is:

[0016]

[0017] Where, t: expected value, t∈R + : expected value of a single optimization target parameter; c: constraint value, c∈R + : Restriction on a single optimization target parameter; o: actual value, o∈R +: The actual value of a certain optimization target parameter of the current quantitative model; Δtj = t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing loss, even if each optimization target parameter is close to the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization objective parameters; j : Penalty factor, λ∈R + ;(max(0,Δ cj )) 2 : Penalty term: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirement; M: The total number of the optimization target parameters. Through this embodiment, a preferred loss function algorithm is provided.

[0018] Optionally, updating the quantization parameters based on the calculated loss function includes: determining and recording a new set of quantization parameters of the second model based on the input function value and the target algorithm; replacing the current quantization parameters of the second model with the new set of quantization parameters; wherein the target algorithm includes a Bayesian optimization algorithm. Through this embodiment, the quantization parameters can be quickly updated.

[0019] Optionally, in response to the operation steps of the loop being executed satisfying the preset condition, obtaining the optimal quantization parameter includes: when the preset condition is satisfied, screening the recorded multiple groups of quantization parameters, a group that can minimize the output function value of the loss function as the optimal quantization parameter; wherein the preset condition includes the number of iterations satisfying the preset number of iterations or the iteration duration satisfying the preset iteration duration. Through this embodiment, the group with the smallest output function value of the loss function is selected as the most effective quantization parameter, thereby ensuring the accuracy of the optimal quantization parameter.

[0020] Optionally, the precision type corresponding to the quantization parameter includes at least one of the following: INT4, INT8 and INT16.

[0021] Optionally, the quantization method further includes setting a set of initial quantization parameters for the first model during an initialization phase, wherein the precision type of the initial quantization parameters includes at least one of the following types: INT4, INT8 and INT16.

[0022] According to another aspect of the exemplary embodiment of the present disclosure, a quantization device for a deep learning model is provided, including: a mixed precision quantization module, configured to perform quantization on a first model based on quantization parameters to obtain a second model; test the second model to obtain actual values ​​of multiple optimization target parameters; a multi-objective optimization module, configured to calculate a loss function based on the actual values ​​of the multiple optimization target parameters and the expected values ​​of the multiple optimization target parameters and the constraint values ​​of the multiple optimization target parameters; an automatic optimizer module, configured to update the quantization parameters based on the calculated loss function, and use the second model as the first model; in response to the update operation satisfying a preset condition, obtain the optimal quantization parameters, and use the first model quantized based on the optimal quantization parameters as the final quantized model.

[0023] Optionally, the mixed precision quantization module is configured to perform quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; perform quantization configuration on the simulation quantization model based on quantization parameters; calculate quantization coefficients based on the simulation quantization model after quantization configuration; and perform model rewriting on the simulation quantization model based on the quantization coefficients to obtain the second model.

[0024] Optionally, the mixed precision quantization module is configured to: insert a simulation quantization operator into each operator to be quantized, wherein the simulation quantization operator includes a simulation quantization operator for quantization weights and a simulation quantization operator for quantization activation values.

[0025] Optionally, the mixed precision quantization module is configured to: analyze the correspondence between the quantization parameters and the hierarchical order of operators in the simulation quantization model; and configure the quantization parameters to corresponding operators based on the correspondence.

[0026] Optionally, the mixed precision quantization module is configured to: determine the precision type of data corresponding to the quantization parameters configured for each simulation quantization operator inserted in the simulation quantization model; simulate the truncation error and rounding error from floating point numbers to integers based on the mapping relationship between floating point data and data of the precision type, and then calculate the quantization coefficient of each simulation quantization operator.

[0027] Optionally, the mixed precision quantization module is configured to: for each simulation quantization operator in the simulation quantization model, determine the quantization parameters and corresponding quantization coefficients of the simulation quantization operator; and replace the simulation quantization operator with a low-precision operator that supports the quantization parameters and quantization coefficients.

[0028] Optionally, the target parameter includes accuracy and at least one of the following parameters: size, energy consumption and inference latency of the quantization model.

[0029] Optionally, the multi-objective optimization module is configured to calculate a loss function based on differences between expected values ​​of the multiple optimization objective parameters and corresponding actual values ​​and differences between constraint values ​​of the multiple optimization objective parameters and corresponding actual values.

[0030] Optionally, the function expression of the loss function is:

[0031]

[0032] Where, t: expected value, t∈R + : expected value of a single optimization target parameter; c: constraint value, c∈R + : Restriction on a single optimization target parameter; o: actual value, o∈R + : The actual value of a certain optimization target parameter of the current quantitative model; Δtj = t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing loss, even if each optimization target parameter is close to the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization objective parameters; j : Penalty factor, λ∈R + ;(max(0,Δ cj )) 2 : Penalty item: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirements; M: The total number of the optimization target parameters.

[0033] Optionally, the automatic optimizer module is configured to: determine and record a new set of quantization parameters for the second model based on the function value of the output of the loss function and the target algorithm; replace the current quantization parameters of the second model with the new set of quantization parameters; wherein the target algorithm includes a Bayesian optimization algorithm.

[0034] Optionally, the automatic optimizer module is configured to: when a preset condition is met, filter out the recorded multiple groups of quantization parameters, and select the group that can minimize the output function value of the loss function as the optimal quantization parameter; wherein the preset condition includes that the number of iterations satisfies the preset number of iterations or the iteration duration satisfies the preset iteration duration.

[0035] Optionally, the precision type corresponding to the quantization parameter includes at least one of the following: INT4, INT8 and INT16.

[0036] Optionally, the quantization device further includes: an initialization module configured to: in an initialization phase, for the first model, set a set of initial quantization parameters, whose precision types include INT4, INT8 and / or INT16.

[0037] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the quantization method for a deep learning model described in the present disclosure is implemented.

[0038] According to another aspect of the exemplary embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the quantization method for a deep learning model described in the present disclosure.

[0039] By using the technical solution disclosed in the present invention, the actual values ​​of multiple optimization target parameters are obtained by performing a quantization test on the first model loop, and then the loss function is calculated according to the actual values, expected values, and constraint values ​​of the multiple optimization target parameters, and the quantization parameters are updated according to the determined loss function, and finally the optimal quantization parameters are obtained, and then the final quantized model can be configured according to the final quantization parameter. In this process, quantization is performed based on the quantization parameters, so that quantization can be performed for different precision types of quantization parameters, avoiding the problem of quantizing the model to a fixed precision and poor compression effect, and can also ensure the mixing of high-precision quantization and low-precision quantization; in addition, the actual values, expected values, and constraint values ​​of multiple optimization target parameters are combined to obtain the loss function in the optimization process, so that the conflict relationship between the optimization targets can be fully considered and unified optimization can be performed; in addition, the present invention directly quantizes the pre-trained model, and does not complete the quantization in the training stage, avoiding the problem of completing the quantization in the training stage and the slow convergence of the quantization model training. Therefore, compared with the quantized model of the prior art, the quantized model of the present invention has a faster convergence speed, good compression effect of the model, and a mixture of high-precision quantization and low-precision quantization, which can ensure the accuracy of the model without fine-tuning.

[0040] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other objects and features of the exemplary embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate the embodiments, in which:

[0042] Figure 1 is a flowchart illustrating a quantization method for a deep learning model according to an exemplary embodiment of the present disclosure;

[0043] Figure 2 is a block diagram illustrating a quantization apparatus for a deep learning model according to an exemplary embodiment of the present disclosure;

[0044] Figure 3 is a multi-objective automatic quantization flow chart showing a quantization method for a deep learning model according to an exemplary embodiment of the present disclosure;

[0045] Figure 4 is a schematic diagram showing a convolution operator quantization process according to an exemplary embodiment of the present disclosure;

[0046] Figure 5 is a schematic diagram showing a quantization execution according to an exemplary embodiment of the present disclosure;

[0047] Figure 6 is a schematic diagram showing a quantization configuration according to an exemplary embodiment of the present disclosure;

[0048] Figure 7 is a schematic diagram showing a quantization method for a deep learning model according to an exemplary embodiment of the present disclosure, and a model expression fragment after quantizing an initial model. DETAILED DESCRIPTION

[0049] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0050] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0051] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0052] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0053] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0054] According to one aspect of an exemplary embodiment of the present disclosure, a quantization method for a deep learning model is provided. Figure 1 is a flowchart illustrating a quantization method for a deep learning model according to an exemplary embodiment of the present disclosure. Figure 1 The quantization method includes steps S110 to S160 which are executed in a loop until a preset condition is met.

[0055] In step S110 , quantization is performed on the first model based on the quantization parameter to obtain a second model.

[0056] It can be understood that the above step S110 can be performed in the following manner.

[0057] For example, according to Figure 5 , step S110 can be refined into the following four steps A to D. Step A is quantization labeling, step B is quantization configuration, step C is quantization coefficient calculation, and step D is model rewriting.

[0058] Specifically, step A can perform quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; step B can perform quantization configuration on the simulation quantization model based on the quantization parameters; step C can calculate the quantization coefficient based on the simulation quantization model after quantization configuration; step D can perform model rewriting on the simulation quantization model based on the quantization coefficient to obtain the second model.

[0059] It should be noted that the first model may be a pre-trained model, for example Figures 3 to 5The precision types corresponding to the quantization parameters include INT4, INT8 and / or INT16, where INT is integer data Integer.

[0060] Furthermore, step A may be implemented in the following manner, for example, inserting a simulation quantization operator into each operator to be quantized, wherein the simulation quantization operator includes a simulation quantization operator for quantization weights and a simulation quantization operator for quantization activation values.

[0061] Specifically, refer to Figure 4 Taking the convolution operator conv2d as an example, the inserted simQ operator (weight) is the simulation quantization operator for the quantized weight, and the simQ (input) operator is the simulation quantization operator for the quantized activation value.

[0062] Optionally, when quantization is specifically performed, uniform quantization can be performed, that is, the simulation quantization operator for the quantization weights can adopt asymmetric quantization, and the simulation quantization operator for the quantization activation values ​​can adopt symmetric quantization. It should be noted that the simulation quantization operator for the above-mentioned quantization weights can also adopt symmetric quantization, and the simulation quantization operator for the quantization activation values ​​can also adopt asymmetric quantization.

[0063] Step B may be implemented in the following manner, for example, by analyzing the correspondence between the quantization parameters and the hierarchical order of operators in the simulation quantization model; and configuring the quantization parameters to the corresponding operators based on the correspondence.

[0064] The quantization parameters may be generated by an optimizer based on a loss function (described in detail later). Optionally, the correspondence between the operator level in the simulation quantization model and the parameters in the quantization parameter array may be a one-to-one mapping. Figure 6 , the Layers array represents the quantization parameters, and the corresponding quantization parameters are configured in the corresponding operators. Among them, parameters 4, 8, and 16 indicate that the precision type of the calculation data of the corresponding simulation quantization operator is configured as INT4, INT8, and INT16 respectively.

[0065] For example, refer to Figure 4 In the pre-trained model, a convolutional layer conv2d originally performs FP32 operations. Through the quantization of the simulation quantization operator, the FP32 to INT8 conversion is realized, so that the conv2d can perform INT8 calculations, where FP is floating point data. In fact, the role of this step B is to determine the accuracy with which each of the multiple operators (for example, convolutional layers) that perform calculations in the model should perform calculations.

[0066] For step C, it can be implemented in the following manner, for example, determining the precision type of the data corresponding to the quantization parameters configured for each simulation quantization operator inserted in the simulation quantization model; based on the mapping relationship between floating-point data and data of the precision type, simulating the truncation error and rounding error from floating-point numbers to integers, and then calculating the quantization coefficient of each simulation quantization operator.

[0067] Reference Figure 6 The configured quantization parameters can be, for example, [8, 4, ... 16, ... 4, 8, ... , 4, 16, ...], where the value 4 indicates that the precision type of the data corresponding to the configured quantization parameter is INT4. Similarly, the value 8 corresponds to INT8, and the value 16 corresponds to INT16.

[0068] It should be noted that the mapping relationship between floating-point data and data of the precision type can be acquired in advance, which is not the focus of the present disclosure and will not be described in detail.

[0069] It can be understood that the above-mentioned "simulated" truncation error and rounding error can be achieved in the following manner. For example, the activation value can be truncated and rounded using the KLD and Min / Max methods; the weight value can be truncated and rounded using the Max / Min method. Then, the truncation error and rounding error are determined based on the comparison between the truncated value and the rounded value and the original floating-point data.

[0070] Afterwards, the quantization coefficient of each simulation quantization operator can be determined based on the "mapping relationship", "truncation error" and "rounding error". The quantization coefficient can make the data as close to the original data distribution as possible after quantization, wherein the quantization coefficient may include a scale factor and a zero point, for example.

[0071] For step D, it can be implemented in the following manner, for example, for each simulation quantization operator in the simulation quantization model, determine the quantization parameters and corresponding quantization coefficients of the simulation quantization operator; and replace the simulation quantization operator with a low-precision operator that supports the quantization parameters and quantization coefficients.

[0072] It is understood that the above-mentioned "replacement" can be performed in a variety of available ways. For example, referring to Figure 4For example, the Mul, Round, Clip and Cast functions can be used to process the simulation operator to obtain the low-precision operator. Among them, the meaning of the Mul function is: FP32*1 / scale, that is, dividing the floating point number by scale; the Round function is explained by taking the following example, and the meaning of Round(Mul(FP32 / scale)) is: rounding the data; the Clip function is explained by taking the following example, and the meaning of Clip(Round(Mul(FP32 / scale)), -2^(bit-1), +2^(bit-1)-1) is: set the number less than -2^(bit-1) to -2^(bit-1). For example, when the operator is selected as INT4, that is, 4 bits, -2^(bit-1) is -8, and the number greater than +2^(bit-1) is set to +2^(bit-1)-1. When the operator selects the precision INT4, that is, 4 bits, +2^(bit-1)-1 is +7, that is, the number greater than +7 is set to +7; the meaning of the Cast function is to convert FP32 to INT type, (FP32)+7–>(INT4)+7)

[0073] In the above-mentioned quantization process, in order to ensure the accuracy of low-precision quantization, the model is quantized according to the quantization parameters, so that different quantization modes can be adopted according to the precision types of different quantization parameters, such as INT8 and INT16 adopt hierarchical quantization modes, and INT4 adopts channel-level quantization modes. High-precision operators help to ensure accuracy, and low-precision operators help to compress model size and inference delay, avoiding the problem of quantizing the model to a fixed precision and poor compression effect, and also realizing the mixture of high-precision quantization and low-precision quantization, which can ensure the accuracy of the model without fine-tuning. In step S120, the second model is tested to obtain the actual values ​​of multiple optimization target parameters.

[0074] It should be noted here that, refer to Figure 3 , Figure 5 , where Val.Data is a test data set, and the second model is tested using Val.Data. Specifically, the quantized model (ie, the second model) is compiled and predicted to obtain actual values ​​of multiple optimization target parameters.

[0075] The target parameters include: size, accuracy, power and / or inference latency of the quantized model. It should be noted that the target parameters include but are not limited to the above parameters, and may also include other indicators that can characterize the performance of the model.

[0076] In step S130, a loss function is calculated based on actual values ​​of multiple optimization target parameters, expected values ​​of multiple optimization target parameters, and constraint values ​​of multiple optimization target parameters.

[0077] It can be understood that the loss function can be calculated in the following manner. For example, the loss function can be calculated based on the difference between the expected values ​​of the multiple optimization target parameters and the corresponding actual values ​​and the difference between the constraint values ​​of the multiple optimization target parameters and the corresponding actual values, wherein the function expression of the loss function is:

[0078]

[0079] Where, t: expected value, t∈R + : expected value of a single optimization target parameter; c: constraint value, c∈R + : Restriction on a single optimization target parameter; o: actual value, o∈R + : The actual value of a certain optimization target parameter of the current quantitative model; Δtj = t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing loss, even if each optimization target parameter is close to the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization objective parameters; j : Penalty factor, λ∈R + ;(max(0,Δ cj )) 2 : Penalty item: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirements; M: The total number of the optimization target parameters.

[0080] As mentioned above, by integrating the multi-objective expected values ​​and constraint values, the final comprehensive proxy loss function is obtained. The smaller the loss function value loss is, the closer the quantitative model is to the Pareto optimal solution.

[0081] In step S140, the quantization parameter is updated based on the calculated loss function, and the second model is used as the first model.

[0082] Specifically, the update can be achieved in the following ways. For example, the function value of the output of the loss function can be input into the optimizer, and the optimizer performs calculations based on the input function value and the target algorithm (also referred to as the optimization algorithm), determines a new set of quantization parameters for the second model, and records them; and uses the new set of quantization parameters to replace the current quantization parameters of the second model; wherein the target algorithm includes a Bayesian optimization algorithm, and it should be noted that the above-mentioned target algorithm is not limited to Bayesian optimization, and can also be reinforcement learning (abbreviated as RL), genetic algorithm, etc.

[0083] In step S150, in response to the operation steps being executed in a loop satisfying the preset condition, an optimal quantization parameter is obtained, and the first model quantized based on the optimal quantization parameter is used as the final quantized model.

[0084] Before step S150, it may also be determined whether the operation steps to be cyclically executed meet the preset conditions.

[0085] Here, the operation steps of the loop execution are the above-mentioned steps S110 to S140. Specifically, when the preset condition is met, the set of quantization parameters recorded is selected to minimize the output function value of the loss function as the optimal quantization parameter; wherein the preset condition includes that the number of iterations meets the preset number of iterations or the iteration duration meets the preset iteration duration.

[0086] In one embodiment, the quantization method further comprises, during an initialization phase, setting a set of initial quantization parameters for the first model, whose precision types include INT4, INT8 and / or INT16.

[0087] It can be understood that in the initial stage, a set of initial quantization parameters is given to the pre-trained model, which can trigger the above steps S110 to S140 to be executed in a loop until the preset conditions are met. Optionally, the precision type of the initial quantization parameters can all be set to INT8.

[0088] Combine the following Figure 3 and Figure 7 The quantization method for the deep learning model described above is described in the present disclosure:

[0089] Reference Figure 3, the mixed precision quantization module (MP-Quant) executes steps S110 to S120. Specifically, this module mainly converts the FP32 operator corresponding to each layer in the original floating-point model into an integer operator with the precision set by the quantization configuration module, and uses the low-precision computing units such as NPU or GPU to speed up the calculation execution speed. The activation values ​​and weights corresponding to each layer (Convolution layer or Dense layer) are quantized at the same time, and the quantized model is compiled and evaluated to obtain (ModelSize, Acc., etc.) information. There are different ways to quantize according to different precisions (please refer to the relevant content in the embodiments of the invention).

[0090] The multi-objective optimization module (MOO) is mainly used to execute step S130. Specifically, this module is responsible for comprehensively quantifying multiple target values ​​and converting the optimization task into a comprehensive proxy loss function. When the neural network model is actually applied, multiple aspects need to be considered. In addition to the accuracy of the model, the size of the model, inference delay and energy consumption also affect the application of the neural network model in actual scenarios. In actual applications, there are conflicts between multiple goals, such as accuracy and model size: a small model generally leads to low model quantization accuracy, and the comprehensive optimal solution requires weighing the various goals of the model. This module designs a comprehensive proxy loss function as the output of the quantization model to feed back to the optimizer to comprehensively optimize multiple goals (accuracy, model size, inference time, etc.) at the same time.

[0091] The automatic optimizer module (Auto-Opt) is mainly used to execute steps S140 to S160. Specifically, the automatic optimizer module is responsible for selecting the best overall result (optimal Pareto solution). The optimizer optimizes the model to be quantized as a black box function, and treats the precision quantization configuration of each layer of the model as a hyperparameter. The optimizer searches for the optimal Pareto solution through iterative optimization. In each round of iteration, the optimizer accepts the result of the previous round of hyperparameters acting on the model as input, adjusts the optimizer's posterior probability distribution, and then generates new hyperparameters for the next round of iteration. When the number of iterations reaches the pre-selected target, or the pre-set optimization time is exhausted, the optimizer stops optimizing and outputs the strategy of the optimal Pareto solution, that is, the optimal mixed precision configuration (optimal quantization parameters). Among them, Figure 3 The Config Space shown in is mainly used to provide a screening data set for the optimizer.

[0092] like Figure 3 As described above, when the loop optimization iteration reaches the preset condition, the best set of quantization parameters is selected as the best quantization strategy (Best QStrategy), and then the model is quantized based on the best quantization strategy to obtain the final quantization model.

[0093] Combination Figure 7The quantized model expression fragment shown is for the convolution operator conv2d, which is converted from the corresponding FP3 floating point operator to an integer operator with INT4 and INT8 precision. The specific conversion process steps refer to steps A to D in the aforementioned step S110, which will not be repeated here.

[0094] According to another aspect of the exemplary embodiment of the present disclosure, a quantization device for a deep learning model is provided. Figure 2 The quantization device 200 includes a mixed precision quantization module 210, a multi-objective optimization module 220, an automatic optimizer module 230 and an initialization module 240, and each unit can be coupled to each other in a communicative manner.

[0095] The mixed precision quantization module 210 can be configured to perform quantization on the first model based on the quantization parameters to obtain the second model; test the second model to obtain the actual values ​​of multiple optimization target parameters; the multi-objective optimization module 220 is configured to calculate the loss function based on the actual values ​​of multiple optimization target parameters and the expected values ​​of multiple optimization target parameters and the constraint values ​​of multiple optimization target parameters; the automatic optimizer module 230 is configured to update the quantization parameters based on the calculated loss function and use the second model as the first model; in response to the update operation satisfying the preset conditions, the optimal quantization parameters are obtained, and the first model quantized based on the optimal quantization parameters is used as the final quantized model.

[0096] It should be understood that the specific features described in the aforementioned quantization method for deep learning models of the present disclosure can also be similarly applied to the quantization device of the deep learning model for similar expansion. For the sake of simplicity, it is not described in detail.

[0097] Optionally, the mixed precision quantization module is configured to perform quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; perform quantization configuration on the simulation quantization model based on quantization parameters; calculate quantization coefficients based on the simulation quantization model after quantization configuration; and perform model rewriting on the simulation quantization model based on the quantization coefficients to obtain the second model.

[0098] Optionally, the mixed precision quantization module is configured to insert a simulation quantization operator into each operator to be quantized, wherein the simulation quantization operator includes a simulation quantization operator for quantization weights and a simulation quantization operator for quantization activation values.

[0099] Optionally, the mixed precision quantization module is configured to analyze the correspondence between the quantization parameters and the hierarchical order of operators in the simulation quantization model; and configure the quantization parameters to corresponding operators based on the correspondence.

[0100] Optionally, the mixed precision quantization module is configured to determine the precision type of data corresponding to the quantization parameters configured for each simulation quantization operator inserted in the simulation quantization model; based on the mapping relationship between floating-point data and data of the precision type, simulate the truncation error and rounding error from floating-point numbers to integers, and then calculate the quantization coefficient of each simulation quantization operator.

[0101] Optionally, the mixed precision quantization module is configured to determine, for each simulation quantization operator in the simulation quantization model, the quantization parameters and corresponding quantization coefficients of the simulation quantization operator to be quantized; and replace the simulation quantization operator with a low-precision operator that supports the quantization parameters and quantization coefficients.

[0102] Optionally, the target parameter includes accuracy and at least one of the following parameters: size, energy consumption and inference latency of the quantization model.

[0103] Optionally, the multi-objective optimization module is configured to calculate a loss function based on the difference between the expected values ​​of the multiple optimization objective parameters and the corresponding actual values ​​and the difference between the constraint values ​​of the multiple optimization objective parameters and the corresponding actual values. The function expression of the loss function is:

[0104]

[0105] Where, t: expected value, t∈R + : expected value of a single optimization target parameter; c: constraint value, c∈R + : Restriction on a single optimization target parameter; o: actual value, o∈R + : The actual value of a certain optimization target parameter of the current quantitative model; Δtj = t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing loss, even if each optimization target parameter is close to the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization objective parameters; j : Penalty factor, λ∈R + ;(max(0,Δ cj )) 2: Penalty item: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirements; M: The total number of the optimization target parameters.

[0106] Optionally, the automatic optimizer module is configured to determine and record a new set of quantization parameters for the second model based on the function value of the output of the loss function and the target algorithm; and replace the current quantization parameters of the second model with the new set of quantization parameters; wherein the target algorithm includes a Bayesian optimization algorithm.

[0107] Optionally, the automatic optimizer module is configured to filter out the recorded multiple groups of quantization parameters, and select a group that can minimize the output function value of the loss function as the optimal quantization parameter when a preset condition is met; wherein the preset condition includes that the number of iterations satisfies a preset number of iterations or the iteration duration satisfies a preset iteration duration.

[0108] Optionally, the precision type corresponding to the quantization parameter includes at least one of the following: INT4, INT8 and INT16.

[0109] Optionally, the quantization device further includes: an initialization module 240, configured to set a set of initial quantization parameters for the first model during an initialization phase, and the precision types thereof include INT4, INT8 and / or INT16.

[0110] It should be understood that each unit / module in the quantization system for deep learning models according to the exemplary embodiments of the present disclosure can be implemented as a hardware component and / or a software component. Those skilled in the art can implement each unit / module, for example, using a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), according to the processing performed by each defined unit / module.

[0111] According to another aspect of the exemplary embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the quantization method for a deep learning model described in the present disclosure is implemented.

[0112] Specifically, the quantization method for a deep learning model according to an exemplary embodiment of the present disclosure may be written as a computer program, a code segment, an instruction, or any combination thereof, and recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. The computer-readable storage medium is any data storage device that can store data read out by a computer system. Examples of computer-readable storage media include: read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, optical data storage device, and carrier wave (such as data transmission over the Internet via a wired or wireless transmission path).

[0113] According to another aspect of the exemplary embodiments of the present disclosure, an electronic device is provided, wherein the electronic device includes: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to execute the quantization method for a deep learning model described in the present disclosure.

[0114] Specifically, the electronic device can be broadly a tablet computer, a smart phone, a smart watch, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the electronic device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the electronic device can be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device can be used to connect and communicate with external devices via a network.

Claims

1. A quantization method for a deep learning model executed by a computer, the computer comprising a memory and a processor, the memory storing computer executable instructions, when the computer executable instructions are executed by the processor, the quantization method executed comprises: The following steps are repeated until the preset conditions are met: quantize the first model based on the quantization parameter to obtain a second model, wherein input data of the first model includes image data; Testing the second model to obtain actual values ​​of multiple optimization target parameters; Calculating a loss function based on actual values ​​of multiple optimization target parameters, expected values ​​of multiple optimization target parameters, and constraint values ​​of multiple optimization target parameters; Based on the calculated loss function, the quantization parameter is updated, and the second model is used as the first model; In response to the operation steps being executed in a loop satisfying the preset condition, an optimal quantization parameter is obtained, and a first model quantized based on the optimal quantization parameter is used as a final quantized model.

2. The quantization method according to claim 1, characterized in that: The step of performing quantization on the first model based on the quantization parameter to obtain the second model comprises: Performing quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; Based on the quantization parameters, performing quantization configuration on the simulation quantization model; Calculate the quantization coefficient based on the simulation quantization model after quantization configuration; Model rewriting is performed on the simulated quantization model based on the quantization coefficients to obtain the second model.

3. The quantization method according to claim 1, characterized in that: The target parameter includes accuracy and at least one of the following parameters: size of a quantization model, energy consumption, and inference latency.

4. The quantization method according to claim 1, characterized in that: Based on the actual values ​​of multiple optimization target parameters, the expected values ​​of multiple optimization target parameters, and the constraint values ​​of multiple optimization target parameters, the loss function is calculated, including: A loss function is calculated based on the differences between the expected values ​​of the plurality of optimization objective parameters and the corresponding actual values ​​and the differences between the constraint values ​​of the plurality of optimization objective parameters and the corresponding actual values.

5. The quantization method according to claim 4, characterized in that: The functional expression of the loss function is: Where, t: expected value, t∈R + : The expected value of a single optimization target parameter; c: constraint value, c∈R + : Restrictions on a single optimization target parameter; o: actual value, o∈R + : The actual value of a certain optimization target parameter of the current quantitative model; Δtj=t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing the loss, even if each optimization target parameter approaches the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization target parameters; λ j : Penalty factor, λ∈R + ; (max(0,Δ cj )) 2 : Penalty term: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirements; M: the total number of optimization target parameters.

6. The quantization method according to claim 1, characterized in that: Based on the calculated loss function, the quantization parameter is updated, including: Determine and record a new set of quantization parameters of the second model based on the function value of the output of the loss function and the target algorithm; The new set of quantization parameters is used to replace the current quantization parameters of the second model; wherein the target algorithm includes a Bayesian optimization algorithm.

7. The quantization method according to claim 6, characterized in that: In response to the operation steps being executed in a loop satisfying the preset condition, obtaining an optimal quantization parameter includes: When the preset conditions are met, a group of the recorded quantization parameters that can minimize the output function value of the loss function is selected as the optimal quantization parameter; The preset conditions include that the number of iterations meets the preset number of iterations or the iteration duration meets the preset iteration duration.

8. The quantization method according to claim 1, characterized in that: The precision type corresponding to the quantization parameter includes at least one of the following: INT4, INT8 and INT16.

9. A quantization device for a deep learning model, the quantization device for a deep learning model is applied on a computer, the computer includes a memory and a processor, the memory stores computer executable instructions, when the computer executable instructions are executed by the processor, the quantization device includes: The mixed precision quantization module is configured as: quantize the first model based on the quantization parameter to obtain a second model, wherein input data of the first model includes image data; Testing the second model to obtain actual values ​​of multiple optimization target parameters; The multi-objective optimization module is configured as follows: Calculating a loss function based on actual values ​​of multiple optimization target parameters, expected values ​​of multiple optimization target parameters, and constraint values ​​of multiple optimization target parameters; The automatic optimizer module is configured to: Based on the calculated loss function, the quantization parameter is updated, and the second model is used as the first model; In response to the updating operation satisfying a preset condition, an optimal quantization parameter is obtained, and the first model quantized based on the optimal quantization parameter is used as a final quantized model.

10. The quantization device according to claim 9, characterized in that: The mixed precision quantization module is configured as follows: Performing quantization labeling on each operator to be quantized in the first model to obtain a simulation quantization model; Based on the quantization parameters, performing quantization configuration on the simulation quantization model; Calculate the quantization coefficient based on the simulation quantization model after quantization configuration; Model rewriting is performed on the simulated quantization model based on the quantization coefficients to obtain the second model.

11. The quantization device according to claim 9, characterized in that: The target parameter includes accuracy and at least one of the following parameters: size of a quantization model, energy consumption, and inference latency.

12. The quantization device according to claim 9, characterized in that: The multi-objective optimization module is configured as follows: A loss function is calculated based on the differences between the expected values ​​of the plurality of optimization objective parameters and the corresponding actual values ​​and the differences between the constraint values ​​of the plurality of optimization objective parameters and the corresponding actual values.

13. The quantization device according to claim 12, characterized in that: The functional expression of the loss function is: Where, t: expected value, t∈R + : The expected value of a single optimization target parameter; c: constraint value, c∈R + : Restrictions on a single optimization target parameter; o: actual value, o∈R + : The actual value of a certain optimization target parameter of the current quantitative model; Δtj=t j -o j : The difference between the actual value and the expected value; Δ cj =c j -o j : The difference between the actual value and the constraint value; w j : Weight factor, w∈R + , Δ tj 2 : Optimization item. When minimizing the loss, even if each optimization target parameter approaches the expected value, the importance of each optimization target parameter is adjusted by the weight factor; w j ×Δ tj 2 The term makes the final result weigh the various optimization target parameters; λ j : Penalty factor, λ∈R + ; (max(0,Δ cj )) 2 : Penalty term: When the actual value of an optimization target parameter of the second model exceeds the limit, it will be penalized so that each optimization target parameter can meet the constraint requirements; M: the total number of optimization target parameters.

14. The quantization device according to claim 9, characterized in that: The automatic optimizer module is configured to: Determine and record a new set of quantization parameters of the second model based on the function value of the output of the loss function and the target algorithm; The new set of quantization parameters is used to replace the current quantization parameters of the second model; wherein the target algorithm includes a Bayesian optimization algorithm.

15. The quantization device according to claim 14, characterized in that: The automatic optimizer module is configured to: When the preset conditions are met, a group of the recorded quantization parameters that can minimize the output function value of the loss function is selected as the optimal quantization parameter; The preset conditions include that the number of iterations meets the preset number of iterations or the iteration duration meets the preset iteration duration.

16. The quantization device according to claim 9, characterized in that: The precision type corresponding to the quantization parameter includes at least one of the following: INT4, INT8 and INT16.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the quantization method for a deep learning model as described in any one of claims 1 to 8 is implemented.

18. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the quantization method for the deep learning model as described in any one of claims 1 to 8.

Images