Quantization method, device, electronic device, storage medium and computer program product

Abstract

The present invention provides a quantization method, device, electronic device, storage medium and computer program product, which relate to the field of natural language technology. The quantization method comprises: for each weight to be quantized in a dialogue model, Gaussian fitting is performed on the weight to be quantized to obtain a Gaussian distribution function; based on the Gaussian distribution function, a quantization parameter value range of linear quantization is determined; maximum likelihood estimation is performed on the estimated parameter variables within the quantization parameter value range of linear quantization to obtain a target threshold; and the target threshold is used to perform a quantization operation on the weight to be quantized. The present invention can significantly reduce the precision loss in the quantization process without using dialogue data set reasoning, thereby improving the accuracy of the dialogue.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a quantization method, device, electronic device, storage medium and computer program product. Background Art

[0002] When building a conversation model through a deep learning (DL) framework, the weight parameters usually default to 32-bit floating point numbers or 16-bit floating point numbers. However, on mobile embedded devices, due to limitations such as memory size and power consumption, quantization methods are usually used to compress the original floating point model to reduce the memory consumption required for the weight parameters.

[0003] Currently, quantization methods include unsaturated mapping quantization method and saturated mapping quantization method. When using the unsaturated mapping quantization method, the weight information near the zero point is seriously lost, resulting in a serious loss of precision during the quantization process, which in turn reduces the accuracy of the dialogue. When using the saturated mapping quantization method, it is necessary to use the dialogue dataset for inference. Summary of the invention

[0004] The present invention provides a quantization method, device, electronic device, storage medium and computer program product, which are used to significantly reduce the precision loss in the quantization process without using dialogue data set reasoning, thereby improving the accuracy of the dialogue.

[0005] The present invention provides a quantification method, comprising:

[0006] For each weight to be quantized in the dialogue model, Gaussian fitting is performed on the weight to be quantized to obtain a Gaussian distribution function;

[0007] Determine a quantization parameter value range of linear quantization based on the Gaussian distribution function;

[0008] Performing maximum likelihood estimation on the estimated parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold;

[0009] The target threshold is used to perform a quantization operation on the weight to be quantized.

[0010] According to a quantization method provided by the present invention, performing maximum likelihood estimation on the estimated parameter variables within the quantization parameter value range of the linear quantization to obtain a target threshold value includes:

[0011] Taking the estimated parameter variable within the range of the quantization parameter of the linear quantization as the independent variable and the similarity variable between the observed data and the expected result as the dependent variable, a joint probability density function is constructed;

[0012] The data within the quantization parameter value range of the linear quantization is traversed by the following steps:

[0013] Selecting a current maximum value and a current minimum value from the range of quantization parameter values ​​of the linear quantization;

[0014] Based on the current maximum value and the current minimum value, quantize and then dequantize the weight to be quantized;

[0015] The data after quantization and then dequantization is used as the observed data, and the original data before quantization is used as the expected result, and the similarity variable value of the joint probability density function is calculated;

[0016] After traversing all data within the value range of the quantization parameter of the linear quantization, determining the maximum value among the similarity variable values ​​as the maximum likelihood estimation value;

[0017] The estimated parameter variable value corresponding to the maximum likelihood estimate is determined as the target threshold.

[0018] According to a quantization method provided by the present invention, the quantization and then dequantization of the weight to be quantized based on the current maximum value and the current minimum value includes:

[0019] Calculate the quotient of the difference between the current maximum value and the current minimum value and a first value as a first quantization coefficient; the first value is the difference between a power of 2 and 1, and the power is the number of quantization bits;

[0020] Calculate the quotient of the sum of the current maximum value and the current minimum value and a second value as a zero point offset value; the second value is 2;

[0021] Calculating a quotient of a difference between the weight to be quantized and the zero offset value and the first quantization coefficient as a quantized value of the weight to be quantized;

[0022] The sum of the product of the quantized value of the weight to be quantized and the first quantized coefficient and the zero offset value is calculated as the inverse quantized value of the quantized value.

[0023] According to a quantization method provided by the present invention, the quantization and then dequantization of the weight to be quantized based on the current maximum value and the current minimum value includes:

[0024] Calculate the quotient of the maximum value of the absolute value of the current maximum value and the absolute value of the current minimum value and a third value as the second quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1;

[0025] Calculating a quotient of the weight to be quantized and the second quantization coefficient as a quantized value of the weight to be quantized;

[0026] The product of the quantized value of the weight to be quantized and the second quantized coefficient is calculated as the inverse quantized value of the quantized value.

[0027] According to a quantization method provided by the present invention, the determining of a quantization parameter value range of linear quantization based on the Gaussian distribution function includes:

[0028] Determine an absolute value of a threshold value corresponding to a variance of the target multiple based on the Gaussian distribution function as a first threshold value;

[0029] Using the positive first threshold as the second threshold and using the negative first threshold as the third threshold;

[0030] Based on the Gaussian distribution function, determining an actual maximum threshold value in an interval between the second threshold value and positive infinity, and determining an actual minimum threshold value in an interval between negative infinity and the third threshold value;

[0031] The value range of the quantization parameter of the linear quantization is determined based on the actual minimum threshold and the actual maximum threshold.

[0032] According to a quantization method provided by the present invention, the absolute value of the threshold value corresponding to the variance of the target multiple determined based on the Gaussian distribution function as the first threshold value includes:

[0033] An absolute value of a threshold corresponding to twice the variance is determined based on the Gaussian distribution function as the first threshold.

[0034] According to a quantization method provided by the present invention, the absolute value of the threshold value corresponding to the variance of the target multiple determined based on the Gaussian distribution function as the first threshold value includes:

[0035] An absolute value of a threshold corresponding to three times the variance is determined based on the Gaussian distribution function as the first threshold.

[0036] According to a quantization method provided by the present invention, for each weight to be quantized in the dialogue model, Gaussian fitting is performed on the weight to be quantized to obtain a Gaussian distribution function, including:

[0037] For each of the weights to be quantized in the dialogue model, Gaussian fitting is performed on the weights to be quantized based on parameter distribution characteristics of the weights to be quantized to determine mathematical expectation and variance;

[0038] The Gaussian distribution function is determined based on the mathematical expectation and the variance.

[0039] According to a quantization method provided by the present invention, the step of using the target threshold to perform a quantization operation on the weight to be quantized includes:

[0040] Determining a quantization range based on positive and negative said target threshold;

[0041] Mapping the data in the weight to be quantized that exceeds the quantization range to the target threshold;

[0042] Perform linear quantization processing on the data within the quantization range in the weight to be quantized.

[0043] According to a quantization method provided by the present invention, the linear quantization processing of the data within the quantization range in the weight to be quantized includes:

[0044] Calculate the quotient of the difference between the maximum value and the minimum value in the quantization range and the first value as the third quantization coefficient; the first value is the difference between a power of 2 and 1, and the power is the number of quantization bits;

[0045] Calculate the quotient of the sum of the maximum value and the minimum value in the quantization range and a second value as a zero point offset value; the second value is 2;

[0046] The quotient of the difference between the data within the quantization range in the weight to be quantized and the zero offset value and the third quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0047] According to a quantization method provided by the present invention, the linear quantization processing of the data within the quantization range in the weight to be quantized includes:

[0048] Calculate the quotient of the maximum value of the absolute value of the maximum value and the absolute value of the minimum value in the quantization range and a third value as a fourth quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1;

[0049] The quotient of the data within the quantization range in the weight to be quantized and the fourth quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0050] The present invention also provides a quantization device, comprising:

[0051] A Gaussian fitting module, used for performing Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function;

[0052] A range determination module, used to determine a value range of a quantization parameter of linear quantization based on the Gaussian distribution function;

[0053] A maximum likelihood estimation module, used to perform maximum likelihood estimation on the estimated parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold;

[0054] The quantization operation module is used to perform a quantization operation on the weight to be quantized using the target threshold.

[0055] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described quantization methods when executing the computer program.

[0056] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the quantization method described above is implemented.

[0057] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned quantization methods.

[0058] The quantization method, device, electronic device, storage medium and computer program product provided by the present invention firstly perform Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function; then, determine the value range of the quantization parameter of the linear quantization based on the Gaussian distribution function; then, perform maximum likelihood estimation on the estimated parameter variables within the value range of the quantization parameter of the linear quantization to obtain a target threshold; in this way, the characteristics of the weight parameter can be greatly retained and the quantization loss introduced by the weight parameter can be reduced; finally, the target threshold is used to perform quantization operations on the weight to be quantized, which can significantly reduce the precision loss in the quantization process without using the dialogue data set reasoning, thereby improving the accuracy of the dialogue. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a schematic diagram of the FP32 and INT8 mixed reasoning process provided by related technologies.

[0061] Figure 2 It is a flowchart of a quantization method provided by an embodiment of the present invention.

[0062] Figure 3 It is a schematic diagram of Gaussian distribution provided by an embodiment of the present invention.

[0063] Figure 4 It is a schematic diagram of the quantization steps and reasoning process provided by an embodiment of the present invention.

[0064] Figure 5 It is a structural schematic diagram of a quantization device provided by an embodiment of the present invention.

[0065] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] When building a conversation model through a DL framework, the weight parameters usually default to 32-bit floating point numbers or 16-bit floating point numbers. However, on mobile embedded devices, due to limitations such as memory size and power consumption, quantization methods are usually used to compress the original floating point model to reduce the memory consumption required for the weight parameters. For example, quantizing a 32-bit floating point number (float32 bit) to an 8-bit integer (int8 bit) directly reduces the memory required to 1 / 4 of the original.

[0068] Model quantization refers to converting the floating-point algorithm of the dialogue model into fixed-point. Model quantization has the following benefits:

[0069] 1) Reduce model size: For example, by quantizing a 32-bit floating point number to an 8-bit integer, the memory required is directly reduced to 1 / 4 of the original, and the model size can be reduced by up to 75%;

[0070] 2) Reduce storage space: When running large models, it is common to encounter insufficient storage space on the end side. Quantization can reduce storage space.

[0071] Currently, quantization methods include unsaturated mapping quantization methods and saturated mapping quantization methods.

[0072] 1) Definition of unsaturated mapping quantization method

[0073] Common quantization schemes use linear mapping. For example, the minimum and maximum values ​​[min, max] of a 32-bit floating point value (FP32 value) in a weight data are mapped to [-127, 127], and the intermediate values ​​are mapped according to a linear relationship. This mapping relationship is symmetrical and is called no saturation mapping.

[0074] 2) Definition of saturation mapping quantization method

[0075] For floating-point numbers near zero, linear mapping results in a large precision loss, so it causes a relatively large precision loss for some weight parameters. To solve this problem, instead of directly mapping [min, max] to [-127, 127] during quantization, a suitable threshold |T| < max(|max|, |min|) can be found, and [-T, T] is mapped to [-127, 127]. For values exceeding the threshold T, they are directly mapped to T. This mapping relationship is called saturation mapping.

[0076] To measure the rationality of threshold selection, the Kullback-Leibler divergence is defined as the degree of difference between the quantized 8-bit integers and the original 32-bit floating-point numbers.

[0077] The KL divergence can be calculated through expression (1) :

[0078] (1).

[0079] Among them, P represents the true probability distribution, Q represents the predicted probability distribution, and the corresponding probability density functions are respectively expressed as and , represents an event, and the difference degree between the two probability distributions is measured through the log function.

[0080] It can be seen that when P = Q in expression (1), = 0.

[0081] Since there are significant differences in the threshold distributions of different layers of different dialogue model structures, reasonable threshold selection should be carried out according to the specific situations of different layers of different dialogue model structures. Specifically, the maximum and minimum values of each layer can be calculated first, and then different thresholds are traversed for the specific data distribution situation, and the one that makes the KL divergence reach the minimum value is selected. The main part of the activation value distribution (the activation value corresponds to the threshold: ) is retained to calculate the mapping coefficient, and the part exceeding is saved as .

[0082] The quantization process is equivalent to information re-encoding. The difference between fixed-point and floating-point is that: floating-point retains a specific number of significant digits, exponents, and fractions, while fixed-point only retains a specific number of integers and decimals.

[0083] The dynamic ranges and minimum precisions of different data types are shown in Table 1.

[0084] Table 1

[0085]

[0086] As shown in Table 1, compared with FP32, the accuracy and dynamic range of INT8 are significantly lower, and it is still quite challenging to reduce the accuracy of FP32 to INT8.

[0087] Taking the data type FP16 as an example, the final calculation method of a data is:

[0088] (2).

[0089] in, Represents a significant number, represents a decimal.

[0090] like Figure 1 As shown in the figure, the data type of the stored weights is INT8. During inference, the network is rewritten to insert quantization (Quantize) and dequantization (Dequantize) layers. This inference method is a mixed inference of FP32 and INT8.

[0091] In the estimation of the quantization parameters of the linear mapping, we define (Quantized) represents the quantized value. (Real) represents a raw float32 value:

[0092] (3).

[0093] Experiments have shown that bias Actually it is not necessary, you can get:

[0094] (4).

[0095] in, Represents the quantization coefficient or scaling factor.

[0096] 1) Unsaturated mapping quantization process

[0097] Step 1: Extract ±|max| of the weights to be quantized and calculate the quantization parameters.

[0098] Step 2: quantize the FP32 / FP16 weights and store them as INT8, and save the quantization parameters.

[0099] Step 3: Dequantize to FP32 / FP16 during calculation.

[0100] When using the unsaturated mapping quantization method, the weight information near the zero point is severely lost, resulting in a serious loss of accuracy in the quantization process, which in turn reduces the accuracy of the conversation.

[0101] 2) Saturation Mapping Quantization Process

[0102] Step 1: Perform FP32 reasoning on the dialogue dataset.

[0103] Step 2: Collect the activation values ​​of each layer and make a histogram.

[0104] Step 3: Calculate the KL divergence distribution of the activation value inference results of different layers of the network structure, and select the activation value T_active with the smallest KL divergence for each layer.

[0105] Step 4: For the weights that need to be quantized, calculate the quantization parameters according to |T_active|.

[0106] Step 5: quantize the FP32 / FP16 weights and store them as INT8, and save the quantization parameters.

[0107] Step 6: Dequantize to FP32 / FP16 according to the quantization parameters during calculation.

[0108] When using the saturated map quantization method, it is necessary to use the dialogue dataset for inference.

[0109] Based on this, an embodiment of the present invention proposes a quantization method, an apparatus, an electronic device, a storage medium, and a computer program product, which are described in detail below.

[0110] Combine the following Figure 2-Figure 4 The quantification method of the present invention is described.

[0111] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of a quantization method provided by an embodiment of the present invention. Figure 2 As shown, the method may include steps 201 to 204.

[0112] Step 201: For each weight to be quantized in the dialogue model, perform Gaussian fitting on the weight to be quantized to obtain a Gaussian distribution function.

[0113] Specifically, the dialogue model may be a dialogue-type large language model. For each weight to be quantized in the dialogue model, a Gaussian fit is performed on the weight to be quantized based on the parameter distribution characteristics of the weight to be quantized to determine the mathematical expectation and variance; based on the mathematical expectation and variance, a Gaussian distribution function is determined.

[0114] Gaussian distribution function As shown in expression (5):

[0115] (5).

[0116] in, represents the weight variable, represents the mathematical expectation, Represents variance.

[0117] like Figure 3 As shown, the peak of the Gaussian distribution function is located at When, and The left and right sides are step-symmetric, so the Gaussian distribution exists in principle:

[0118] 1) Principle: Values ​​are distributed in The probability of being in is 0.6826;

[0119] 2) Principle: Values ​​are distributed in The probability of being in is 0.9544;

[0120] 3) Principle: Values ​​are distributed in The probability of this is 0.9974.

[0121] Optionally, based on the distribution types of multiple weights to be quantized in the dialogue model, weights to be quantized whose parameter distribution is similar to Gaussian distribution are screened out from various weights to be quantized, and Gaussian fitting is performed on the screened weights to be quantized to obtain a Gaussian distribution function. Since the weight information of uniformly distributed weights near the zero point is not seriously lost, the quantization method of this embodiment can be used only for the weights of Gaussian distribution, which can reduce the amount of data processing.

[0122] Step 202: Determine a value range of a quantization parameter for linear quantization based on a Gaussian distribution function.

[0123] like Figure 3 As shown, Falling The probability of falling outside is less than 5%. Less than 0.3%, the range of quantization parameter values ​​for linear quantization can be calculated based on the Gaussian distribution function .

[0124] In one embodiment, step 202 includes steps 2021 to 2024.

[0125] Step 2021: Determine the absolute value of the threshold corresponding to the variance of the target multiple based on the Gaussian distribution function as the first threshold.

[0126] Optionally, step 2021 includes: determining twice the variance based on the Gaussian distribution function The absolute value of the corresponding threshold is used as the first threshold.

[0127] Optionally, step 2021 includes: determining three times the variance based on the Gaussian distribution function The absolute value of the corresponding threshold is used as the first threshold.

[0128] Below Taking as an example, the determination of the value range of the quantization parameter of linear quantization is described.

[0129] Step 2022: Set the positive first threshold As the second threshold, the negative first threshold - As the third threshold;

[0130] Step 2023: Based on the Gaussian distribution function, in the interval between the second threshold and positive infinity The actual maximum threshold is determined within , in the interval between negative infinity and the third threshold The actual minimum threshold is determined within ;

[0131] Step 2024: Based on the actual minimum threshold and the actual maximum threshold Determine the range of quantization parameters for linear quantization .

[0132] Step 203: Perform maximum likelihood estimation on the estimated parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold.

[0133] In one embodiment, if Figure 4 As shown, step 203 includes the following steps.

[0134] Step 2031: construct a joint probability density function using the estimated parameter variable within the range of quantization parameter values ​​of linear quantization as an independent variable and the similarity variable between the observed data and the expected result as a dependent variable.

[0135] Likelihood function is an important concept in statistics. Probability is the possibility of something happening under certain circumstances, while likelihood is just the opposite. It is to speculate the possible environment (parameters) that produced this result under a certain result, which can be regarded as the inverse function of conditional probability.

[0136] (6).

[0137] in, express The estimated parameter variables within represents the similarity dataset, Represents the total number of data in the similarity dataset, Indicates Similarity variable values, represents the joint probability density function, represents the likelihood function, represents the maximum value of the likelihood function, Represents the maximum value of the joint probability density function.

[0138] The data within the range of quantization parameter values ​​of linear quantization are traversed through the following steps 2032 to 2034.

[0139] Step 2032: Select the current maximum value from the range of quantization parameter values ​​of linear quantization and the current minimum .

[0140] Step 2033: Based on the current maximum value and the current minimum value, quantize and then dequantize the weights to be quantized.

[0141] Optionally, step 2033 includes:

[0142] Calculate the current maximum value and the current minimum The quotient of the difference between ; The first value is the difference between the power of 2 and 1, where the power is the number of quantization bits , It can be 8;

[0143] (7)

[0144] Calculate the current maximum value and the current minimum The sum of the two values ​​and the quotient of the second value is used as the zero point offset value. ; The second value is 2;

[0145] (8)

[0146] Calculate the weights to be quantized Zero point offset value The difference between The quotient of is used as the quantized value of the weight to be quantized ;

[0147] (9)

[0148] Calculate the quantized value of the weight to be quantized With the first quantization coefficient The product of The sum of the quantized values ​​is used as the inverse quantized value .

[0149] (10)

[0150] In this embodiment, the quantization and dequantization process takes into account the zero offset.

[0151] Optionally, step 2033 includes:

[0152] Calculate the absolute value of the current maximum value and the absolute value of the current minimum value The maximum value in The quotient of the third value is taken as the second quantization coefficient ; The third value is the difference between the power of 2 and 1, where the power is the number of quantization bits The difference with 1, It can be 8;

[0153] (11)

[0154] Calculate the weights to be quantized and the second quantization coefficient The quotient of is used as the quantized value of the weight to be quantized ;

[0155] (12)

[0156] Calculate the quantized value of the weight to be quantized and the second quantization coefficient The product of the quantized value is used as the dequantized value .

[0157] (13)

[0158] In this embodiment, the quantization and dequantization process does not take the zero offset into consideration.

[0159] Step 2034: Using the quantized and then dequantized data as observation data and the original data before quantization as the expected result, calculate the similarity variable value of the joint probability density function.

[0160] Step 2035: After traversing all data within the value range of the quantization parameter of the linear quantization, the maximum value among the similarity variable values ​​is determined as the maximum likelihood estimation value.

[0161] (14)

[0162] Step 2036: Determine the estimated parameter variable value corresponding to the maximum likelihood estimate as the target threshold .

[0163] Step 204: Using target threshold Perform quantization operation on the weights to be quantized.

[0164] In one embodiment, step 204 includes: based on the positive and negative target thresholds Determine the quantization range [-T, T]; map the data that exceeds the quantization range [-T, T] in the weight to be quantized to the target threshold ; Perform linear quantization on the data within the quantization range of the weight to be quantized.

[0165] Optionally, linear quantization processing is performed on data within a quantization range in the weight to be quantized, including:

[0166] Calculate the quotient of the difference between the maximum value and the minimum value in the quantization range and the first value as the third quantization coefficient; the first value is the difference between the power of 2 and 1, and the power is the number of quantization bits;

[0167] Calculate the quotient of the sum of the maximum value and the minimum value in the quantization range and the second value as the zero point offset value; the second value is 2;

[0168] The quotient of the difference between the data within the quantization range in the weight to be quantized and the zero point offset value and the third quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0169] In this embodiment, the quantization process takes the zero offset into account.

[0170] Optionally, linear quantization processing is performed on data within a quantization range in the weight to be quantized, including:

[0171] Calculate the quotient of the maximum value of the absolute value of the maximum value and the absolute value of the minimum value in the quantization range and the third value as the fourth quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1;

[0172] The quotient of the data within the quantization range in the weight to be quantized and the fourth quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0173] In this embodiment, the quantization process does not take the zero offset into account.

[0174] In summary, the quantization method provided by the embodiment of the present invention first performs Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function; then, the quantization parameter value range of the linear quantization is determined based on the Gaussian distribution function; then, the estimated parameter variables within the quantization parameter value range of the linear quantization are maximum likelihood estimated to obtain a target threshold; in this way, the characteristics of the weight parameters can be greatly retained and the quantization loss introduced by the weight parameters can be reduced; finally, the target threshold is used to perform a quantization operation on the weights to be quantized, which can significantly reduce the precision loss in the quantization process without using the dialogue data set inference, thereby improving the accuracy of the dialogue.

[0175] The quantization device provided by the present invention is described below. The quantization device described below and the quantization method described above can be referred to each other.

[0176] Please refer to Figure 5 , Figure 5 is a schematic diagram of the structure of a quantization device provided by an embodiment of the present invention. Figure 5 As shown, the device may include:

[0177] A Gaussian fitting module 10 is used to perform Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function;

[0178] A range determination module 20, configured to determine a value range of a quantization parameter of linear quantization based on a Gaussian distribution function;

[0179] The maximum likelihood estimation module 30 is used to perform maximum likelihood estimation on the estimation parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold value;

[0180] The quantization operation module 40 is used to perform a quantization operation on the weights to be quantized using a target threshold.

[0181] In one embodiment, the maximum likelihood estimation module 30 includes:

[0182] A construction unit is used to construct a joint probability density function using an estimated parameter variable within a range of quantization parameter values ​​of linear quantization as an independent variable and a similarity variable between observed data and expected results as a dependent variable;

[0183] The traversal unit is used to traverse the data within the range of the quantization parameter value of the linear quantization through the following steps:

[0184] Selecting a current maximum value and a current minimum value from the range of quantization parameter values ​​of the linear quantization;

[0185] Based on the current maximum value and the current minimum value, the weight to be quantized is quantized and then dequantized;

[0186] The data after quantization and then dequantization is used as the observed data, and the original data before quantization is used as the expected result to calculate the similarity variable value of the joint probability density function;

[0187] A determination unit, configured to determine the maximum value among the similarity variable values ​​as the maximum likelihood estimation value after traversing all data within the value range of the quantization parameter of the linear quantization;

[0188] The estimated parameter variable value corresponding to the maximum likelihood estimate is determined as the target threshold.

[0189] In one embodiment, the traversal unit is specifically used for:

[0190] Calculate the quotient of the difference between the current maximum value and the current minimum value and the first value as the first quantization coefficient; the first value is the difference between the power of 2 and 1, and the power is the number of quantization bits;

[0191] Calculate the quotient of the sum of the current maximum value and the current minimum value and the second value as the zero point offset value; the second value is 2;

[0192] Calculate the quotient of the difference between the weight to be quantized and the zero offset value and the first quantization coefficient as the quantization value of the weight to be quantized;

[0193] The sum of the product of the quantized value of the weight to be quantized and the first quantized coefficient and the zero point offset value is calculated as the inverse quantized value of the quantized value.

[0194] In one embodiment, the traversal unit is specifically used for:

[0195] Calculate the quotient of the maximum value of the absolute value of the current maximum value and the absolute value of the current minimum value and a third value as the second quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1;

[0196] Calculating the quotient of the weight to be quantized and the second quantization coefficient as the quantization value of the weight to be quantized;

[0197] The product of the quantized value of the weight to be quantized and the second quantized coefficient is calculated as the inverse quantized value of the quantized value.

[0198] In one embodiment, the range determination module 20 includes:

[0199] A first determination module is used to determine an absolute value of a threshold value corresponding to the variance of the target multiple based on a Gaussian distribution function as a first threshold value;

[0200] a second determination module, configured to use the positive first threshold as the second threshold and the negative first threshold as the third threshold;

[0201] A third determination module is used to determine the actual maximum threshold value in the interval between the second threshold value and positive infinity, and determine the actual minimum threshold value in the interval between negative infinity and the third threshold value based on the Gaussian distribution function;

[0202] The fourth determination module is used to determine the value range of the quantization parameter of the linear quantization based on the actual minimum threshold and the actual maximum threshold.

[0203] In one embodiment, the first determination module is specifically configured to: determine an absolute value of a threshold corresponding to twice the variance based on a Gaussian distribution function as the first threshold.

[0204] In one embodiment, the first determination module is specifically configured to: determine an absolute value of a threshold corresponding to three times the variance based on a Gaussian distribution function as the first threshold.

[0205] In one embodiment, the Gaussian fitting module 10 is specifically used for:

[0206] For each weight to be quantized in the dialogue model, Gaussian fitting is performed on the weight to be quantized based on the parameter distribution characteristics of the weight to be quantized to determine the mathematical expectation and variance;

[0207] Based on the mathematical expectation and variance, the Gaussian distribution function is determined.

[0208] In one embodiment, the quantization operation module 40 includes:

[0209] a range determination unit, for determining a quantization range based on positive and negative target thresholds;

[0210] A data mapping unit, used for mapping data exceeding the quantization range in the weight to be quantized into a target threshold;

[0211] The linear quantization unit is used to perform linear quantization processing on the data within the quantization range of the weight to be quantized.

[0212] In one embodiment, the linear quantization unit is specifically used for:

[0213] Calculate the quotient of the difference between the maximum value and the minimum value in the quantization range and the first value as the third quantization coefficient; the first value is the difference between the power of 2 and 1, and the power is the number of quantization bits;

[0214] Calculate the quotient of the sum of the maximum value and the minimum value in the quantization range and the second value as the zero point offset value; the second value is 2;

[0215] The quotient of the difference between the data within the quantization range in the weight to be quantized and the zero point offset value and the third quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0216] In one embodiment, the linear quantization unit is specifically used for:

[0217] Calculate the quotient of the maximum value of the absolute value of the maximum value and the absolute value of the minimum value in the quantization range and the third value as the fourth quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1;

[0218] The quotient of the data within the quantization range in the weight to be quantized and the fourth quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

[0219] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the quantization method, which includes: for each weight to be quantized in the dialogue model, performing Gaussian fitting on the weight to be quantized to obtain a Gaussian distribution function; determining a quantization parameter value range of linear quantization based on the Gaussian distribution function; performing maximum likelihood estimation on the estimated parameter variables within the quantization parameter value range of the linear quantization to obtain a target threshold; and performing a quantization operation on the weight to be quantized using the target threshold.

[0220] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0221] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the quantization method provided by the above-mentioned method embodiments, and the method includes: for each weight to be quantized in the dialogue model, performing Gaussian fitting on the weight to be quantized to obtain a Gaussian distribution function; determining a quantization parameter value range for linear quantization based on the Gaussian distribution function; performing maximum likelihood estimation on the estimated parameter variables within the quantization parameter value range for linear quantization to obtain a target threshold; and performing a quantization operation on the weight to be quantized using the target threshold.

[0222] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the quantization method provided by the above-mentioned method embodiments, the method comprising: for each weight to be quantized in the dialogue model, performing Gaussian fitting on the weight to be quantized to obtain a Gaussian distribution function; determining a quantization parameter value range for linear quantization based on the Gaussian distribution function; performing maximum likelihood estimation on the estimated parameter variables within the quantization parameter value range for linear quantization to obtain a target threshold; and performing a quantization operation on the weight to be quantized using the target threshold.

[0223] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0224] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quantization method, characterized in that: include: For each weight to be quantized in the dialogue model, Gaussian fitting is performed on the weight to be quantized to obtain a Gaussian distribution function; Determine a quantization parameter value range of linear quantization based on the Gaussian distribution function; Performing maximum likelihood estimation on the estimated parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold; Performing a quantization operation on the weight to be quantized using the target threshold; The step of determining a quantization parameter value range of linear quantization based on the Gaussian distribution function includes: Determine the absolute value of a threshold value corresponding to the variance of the target multiple based on the Gaussian distribution function as a first threshold value; the target multiple is two or three times; Using the positive first threshold as the second threshold and using the negative first threshold as the third threshold; Based on the Gaussian distribution function, determining an actual maximum threshold value in an interval between the second threshold value and positive infinity, and determining an actual minimum threshold value in an interval between negative infinity and the third threshold value; The value range of the quantization parameter of the linear quantization is determined based on the actual minimum threshold and the actual maximum threshold.

2. The quantization method according to claim 1, characterized in that: The performing maximum likelihood estimation on the estimated parameter variables within the range of the quantization parameter values ​​of the linear quantization to obtain the target threshold value includes: Taking the estimated parameter variable within the range of the quantization parameter of the linear quantization as the independent variable and the similarity variable between the observed data and the expected result as the dependent variable, a joint probability density function is constructed; The data within the quantization parameter value range of the linear quantization is traversed by the following steps: Selecting a current maximum value and a current minimum value from the range of quantization parameter values ​​of the linear quantization; Based on the current maximum value and the current minimum value, quantize and then dequantize the weight to be quantized; The data after quantization and then dequantization is used as the observed data, and the original data before quantization is used as the expected result, and the similarity variable value of the joint probability density function is calculated; After traversing all data within the value range of the quantization parameter of the linear quantization, determining the maximum value among the similarity variable values ​​as the maximum likelihood estimation value; The estimated parameter variable value corresponding to the maximum likelihood estimate is determined as the target threshold.

3. The quantization method according to claim 2, characterized in that: The step of quantizing and then dequantizing the weight to be quantized based on the current maximum value and the current minimum value includes: Calculate the quotient of the difference between the current maximum value and the current minimum value and a first value as a first quantization coefficient; the first value is the difference between a power of 2 and 1, and the power is the number of quantization bits; Calculate the quotient of the sum of the current maximum value and the current minimum value and a second value as a zero point offset value; the second value is 2; Calculating a quotient of a difference between the weight to be quantized and the zero offset value and the first quantization coefficient as a quantized value of the weight to be quantized; The sum of the product of the quantized value of the weight to be quantized and the first quantized coefficient and the zero offset value is calculated as the inverse quantized value of the quantized value.

4. The quantization method according to claim 2, characterized in that: The step of quantizing and then dequantizing the weight to be quantized based on the current maximum value and the current minimum value includes: Calculate the quotient of the maximum value of the absolute value of the current maximum value and the absolute value of the current minimum value and a third value as the second quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1; Calculating a quotient of the weight to be quantized and the second quantization coefficient as a quantized value of the weight to be quantized; The product of the quantized value of the weight to be quantized and the second quantized coefficient is calculated as the inverse quantized value of the quantized value.

5. The quantization method according to claim 1, characterized in that: The step of performing Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function includes: For each of the weights to be quantized in the dialogue model, Gaussian fitting is performed on the weights to be quantized based on parameter distribution characteristics of the weights to be quantized to determine mathematical expectation and variance; The Gaussian distribution function is determined based on the mathematical expectation and the variance.

6. The quantization method according to claim 1, characterized in that: The step of performing a quantization operation on the weight to be quantized by using the target threshold comprises: Determining a quantization range based on positive and negative said target threshold; Mapping the data in the weight to be quantized that exceeds the quantization range to the target threshold; Perform linear quantization processing on the data within the quantization range in the weight to be quantized.

7. The quantization method according to claim 6, characterized in that: The linear quantization processing of the data within the quantization range in the weight to be quantized includes: Calculate the quotient of the difference between the maximum value and the minimum value in the quantization range and the first value as the third quantization coefficient; the first value is the difference between a power of 2 and 1, and the power is the number of quantization bits; Calculate the quotient of the sum of the maximum value and the minimum value in the quantization range and a second value as a zero point offset value; the second value is 2; The quotient of the difference between the data within the quantization range in the weight to be quantized and the zero offset value and the third quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

8. The quantization method according to claim 6, characterized in that: The linear quantization processing of the data within the quantization range in the weight to be quantized includes: Calculate the quotient of the maximum value of the absolute value of the maximum value and the absolute value of the minimum value in the quantization range and a third value as a fourth quantization coefficient; the third value is the difference between a power of 2 and 1, and the power is the difference between the number of quantization bits and 1; The quotient of the data within the quantization range in the weight to be quantized and the fourth quantization coefficient is calculated as the quantization value of the data within the quantization range in the weight to be quantized.

9. A quantization device, characterized in that: include: A Gaussian fitting module, used for performing Gaussian fitting on each weight to be quantized in the dialogue model to obtain a Gaussian distribution function; A range determination module, used to determine a value range of a quantization parameter of linear quantization based on the Gaussian distribution function; A maximum likelihood estimation module, used to perform maximum likelihood estimation on the estimated parameter variables within the range of quantization parameter values ​​of the linear quantization to obtain a target threshold; A quantization operation module, used to perform a quantization operation on the weight to be quantized using the target threshold; Wherein, the range determination module includes: A first determination module is used to determine an absolute value of a threshold value corresponding to a variance of a target multiple based on the Gaussian distribution function as a first threshold value; the target multiple is two or three times; a second determining module, configured to use the positive first threshold as a second threshold and the negative first threshold as a third threshold; A third determination module is used to determine, based on the Gaussian distribution function, an actual maximum threshold value in an interval between the second threshold value and positive infinity, and to determine an actual minimum threshold value in an interval between negative infinity and the third threshold value; The fourth determination module is used to determine the value range of the quantization parameter of the linear quantization based on the actual minimum threshold and the actual maximum threshold.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the quantization method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the quantization method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the quantization method according to any one of claims 1 to 8 is implemented.

Images