A Model Quantization Method, Device and Equipment Based on Optimized KL Divergence

Abstract

An embodiment of the present application provides a model quantization method, device, electronic device, computer-readable storage medium, and computer program product based on optimized KL divergence, which relates to the field of artificial intelligence technology. The method includes: first obtaining an initial KL divergence set of any layer in the original model, and then performing denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to any layer. Since the optimized KL divergence set is obtained after denoising by the preset filtering algorithm, the KL divergence values corresponding to adjacent truncation thresholds do not differ much, which conforms to the principle that the distributions corresponding to adjacent truncation thresholds are extremely similar. On this basis, the target truncation threshold determined according to the optimized KL divergence set is more accurate. If the original model is quantized based on this method, the processing effect of the quantized model is not much different from that of the original model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology. Specifically, the present application relates to a model quantization method, device, electronic device, computer-readable storage medium, and computer program product based on optimized KL divergence. Background Art

[0002] Generally, when quantizing a floating-point model into an int8 model based on a KL-based quantization scheme, the following steps are included: setting different truncation thresholds, calculating the KL divergence value between the quantized output distribution and the original distribution under the truncation threshold, and the KL divergence value can measure the difference between the quantized distribution and the original distribution. Based on the KL divergence value, the truncation threshold corresponding to the smallest KL divergence value is selected as the optimal truncation threshold.

[0003] However, for the KL-based quantization scheme, its core lies in calculating the KL divergence values under different truncation thresholds. For outputs with special distributions, the obtained KL divergence distribution cannot accurately reflect the difference between the quantized distribution and the original distribution.

[0004] Therefore, the KL divergence distribution determined based on the existing scheme cannot accurately reflect the difference between the quantized distribution and the original distribution, resulting in an incorrect quantized value. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to solve the problem that the KL divergence distribution determined based on the existing scheme cannot accurately reflect the difference between the quantized distribution and the original distribution.

[0006] According to one aspect of the embodiments of the present application, a model quantization method based on optimized KL divergence is provided. The method includes:

[0007] Obtaining an initial KL divergence set of any layer in the original model, where the initial KL divergence set includes initial KL divergence values corresponding one-to-one to each truncation threshold in a preset truncation threshold set, and any initial KL divergence value is determined by the following probability distributions: the original probability distribution of the output data set of the layer, the probability distribution of the new output data set determined by the corresponding truncation threshold, and the probability distribution of the new output data set is obtained according to the target quantization type;

[0008] Performing denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to the layer;

[0009] Determining the output quantization value of the layer according to the target truncation threshold of the layer for quantization, where the target truncation threshold is the truncation threshold corresponding to the smallest KL divergence value in the optimized KL divergence set.

[0010] In a possible implementation, obtaining the initial KL divergence set of any layer in the original model includes:

[0011] Input the calibration matrix into the original model for processing to obtain the output data set of each layer of the original model and the original probability distribution of the corresponding output data set;

[0012] Perform the following operations on the output data set according to each truncation threshold pair in the preset truncation threshold set:

[0013] Truncate the output data set of the any layer according to the truncation threshold pair to obtain a new output data set;

[0014] Quantize the new output data set according to the target quantization type and determine the probability distribution of the new output data set;

[0015] Calculate the initial KL divergence value corresponding to the truncation threshold according to the probability distribution of the new output data set and the original probability distribution corresponding to the truncation threshold;

[0016] Determine the initial KL divergence sets corresponding to all truncation thresholds as the initial KL divergence set of the any layer.

[0017] In another possible implementation, the process of obtaining the original probability distribution of the output data set of the any layer includes:

[0018] After obtaining the output data set of the any layer, screen the maximum absolute value of the output data set of the any layer;

[0019] Determine the original probability distribution of the output data set of the any layer according to the maximum absolute value.

[0020] In another possible implementation, denoise the initial KL divergence set according to a preset filtering algorithm to obtain the optimized KL divergence set corresponding to the any layer, including:

[0021] Denoise the initial KL divergence set according to the one-dimensional Gaussian filtering algorithm to obtain the optimized KL divergence set.

[0022] In another possible implementation, the any layer includes shared weights, and the method further includes:

[0023] Determine the shared quantization value of the shared weights of the any layer according to the target quantization type.

[0024] In another possible implementation, after determining the output quantization value and the shared quantization value of the any layer, the method further includes:

[0025] Determine the output quantization value and the shared quantization value of each layer in all layers except the any layer in the original model;

[0026] Quantize the original model according to the output quantization values and shared quantization values of each layer in the original model layer.

[0027] According to another aspect of the embodiments of the present application, there is provided a model quantization device based on optimized KL divergence, and the device includes:

[0028] An acquisition module, configured to acquire an initial KL divergence set of any layer in the original model, where the initial KL divergence set includes initial KL divergence values corresponding one by one to each truncation threshold in a preset truncation threshold set, and any initial KL divergence value is determined by the following probability distributions: the original probability distribution of the output data set of the layer, the probability distribution of a new output data set determined by the corresponding truncation threshold, and the probability distribution of the new output data set is obtained according to the target quantization type;

[0029] A processing module, configured to perform denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to the layer;

[0030] A determination module, configured to determine the output quantization value of the layer according to the target truncation threshold of the layer to quantize the layer, where the target truncation threshold is the truncation threshold corresponding to the smallest KL divergence value in the optimized KL divergence set.

[0031] According to yet another aspect of the embodiments of the present application, there is provided an electronic device, and the electronic device includes: a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the model quantization method based on optimized KL divergence shown in the present application.

[0032] According to yet another aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the model quantization method based on optimized KL divergence shown in the present application are implemented.

[0033] According to yet another aspect of the embodiments of the present application, there is provided a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the model quantization method based on optimized KL divergence shown in the present application are implemented.

[0034] The beneficial effects brought by the technical solutions provided by the embodiments of the present application are:

[0035] The method provided by the embodiment of the present application first obtains the initial KL divergence set of any layer in the original model, and then performs denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain the optimized KL divergence set corresponding to any layer. Since the optimized KL divergence set is obtained after denoising by the preset filtering algorithm, the KL divergence values corresponding to adjacent truncation thresholds do not differ much, which conforms to the principle that the distributions corresponding to adjacent truncation thresholds are extremely similar. On this basis, the target truncation threshold determined according to the optimized KL divergence set is more accurate. If the original model is quantized based on this method, the processing effect of the quantized model is not much different from that of the original model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for description in the embodiments of the present application will be briefly introduced below.

[0037] Figure 1a It is a histogram of dataset A provided by the embodiment of the present application;

[0038] Figure 1b It is a probability distribution diagram of dataset A provided by the embodiment of the present application;

[0039] Figure 2a It is a histogram corresponding to the output dataset of any layer of GoogLeNet provided by the embodiment of the present application;

[0040] Figure 2b It is a KL divergence curve diagram corresponding to the output dataset of any layer of GoogLeNet provided by the embodiment of the present application;

[0041] Figure 3 It is an optimized KL divergence curve diagram provided by the embodiment of the present application;

[0042] Figure 4 It is a flowchart of a model quantization method based on optimized KL divergence provided by the embodiment of the present application;

[0043] Figure 5 It is a structural schematic diagram of a model quantization device based on optimized KL divergence provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The embodiments of the present application will be described below with reference to the drawings in the present application. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation to the technical solutions of the embodiments of the present application.

[0045] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the terms "include" and "comprise" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude the implementation of other features, information, data, steps, operations, elements, components, and / or their combinations supported by the technical field of the present application. It should be understood that when we say an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term. For example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0046] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0047] First, introduce and explain several terms related to the present application:

[0048] The acc metric is used to describe the ratio of the number of correctly predicted positive examples and correctly predicted negative examples to the total number of predictions, and is generally used to evaluate the performance of machine learning. In the implementation of the present application, it can be used to evaluate the performance of the quantized model and the original model respectively. For example, TP: the number of correctly predicted positive examples, FP: the number of incorrectly predicted negative examples, TN: the number of correctly predicted negative examples, FN: the number of incorrectly predicted positive examples. Then the formula for calculating acc is as follows:

[0049]

[0050] First, provide a process for binning the dataset A and determining the probability distribution. The dataset A includes: -1.8, -1, 0.1, 1, 2.1, 2.5, 3, 5, 6.5, 8. The maximum absolute value of the dataset A is 8. According to the range of 0 to 8, the dataset is divided into 4 bins, and the width of each bin is 2.

[0051] The range of bin 1 is: [0, 2), including: -1.8, -1, 0.1, 1, a total of 4 data;

[0052] The range of bin 2 is: [2, 4), including: 2.1, 2.5, 3, a total of 3 data;

[0053] The range of 3 bins is: [4, 6), including: 5, a total of 1 data;

[0054] The range of 4 bins is: [6, 8], including: 6.5, 8, a total of 2 data;

[0055] Then the probabilities corresponding to 1 bin to 4 bins are: 40%, 30%, 10%, 20% respectively. Then the histogram of dataset A is Figure 1a , and the probability distribution diagram of dataset A is Figure 1b .

[0056] It should be noted that the process of determining the probability distribution of a dataset based on N bins involved in the embodiments of the present application can refer to the above process. In the embodiments of the present application, bins can be referred to as blocks.

[0057] Gaussian filtering: including one-dimensional Gaussian filtering and two-dimensional Gaussian filtering, which is used for the process of weighted averaging of an image or a one-dimensional array. Specifically, the value of each pixel point on the image is obtained by weighted averaging of itself and other pixel values in the neighborhood, or a certain value in the one-dimensional array and other adjacent values are obtained by weighted averaging. In the embodiments of the present application, one-dimensional Gaussian filtering is mainly used to perform Gaussian filtering processing on a one-dimensional array. The one-dimensional Gaussian function used in one-dimensional Gaussian filtering is as follows:

[0058]

[0059] float32 - type model: The input and output data of each layer of this model are stored in float32 type. Its advantage is that more information can be retained during the processing process, and its disadvantage is that it occupies too much memory resources and is not very suitable for multi - layer models.

[0060] int8 - type model: The input and output data of each layer of this model are stored in Int8 type. Compared with the float32 - type storage, the retained information is limited, but the occupied memory resources are not much, which is very suitable for multi - layer models.

[0061] In the prior art, in some scenarios, it is necessary to quantize a float32 - type model into an int8 - type model. Therefore, how to quantize a float32 - type model into an int8 - type model has become a difficult problem in the prior art.

[0062] KL divergence: The Kullback-Leibler divergence, also known as relative entropy, is an asymmetric measure of the difference between two probability distributions. In information theory, relative entropy is equivalent to the interpolation of the information entropies of two probability distributions. The KL divergence can be used in some optimization algorithms, such as the expectation-maximization algorithm. In this case, one of the probability distributions involved in the calculation is the true distribution, and the other is the theoretical distribution. The relative entropy represents the information loss generated when using the theoretical distribution to fit the true distribution. Suppose P and Q are two probability distributions P and Q on the random variable X, where P is the true distribution of the random variable X and Q is the fitted distribution of the random variable. Then, in the case of discrete variables, the definition of the KL divergence is as follows:

[0063]

[0064] The KL divergence can measure the distance between two random distributions. When the two random distributions are the same, their KL divergence value is zero. When the difference between the two random distributions increases, their KL divergence value also increases. In specific applications, the KL divergence can be used to compare the similarity of texts.

[0065] Continuing the description in the background art, in the embodiments of the present application, the KL divergence is used to judge the difference between the fitted distribution and the true distribution corresponding to different truncation thresholds. Since for the output with a special distribution, the obtained KL divergence distribution cannot accurately reflect the difference between the fitted distribution and the true distribution. Specifically, the KL divergence distribution curve is not smooth, and for a concentrated distribution, there is a high probability of obtaining a smaller truncation threshold as the optimal truncation threshold.

[0066] The prior art provides an original method for quantifying a neural network based on the KL divergence, which is used to quantify GoogLeNet (a deep neural network model based on the Inception module launched by Google). During the process of quantifying a certain layer of GoogLeNet, a histogram corresponding to the output data set of this layer is obtained. See Figure 2a as shown. Among them, the abscissa is multiple bins, and the ordinate is the number of data in each bin. There are more values near the maximum absolute value, so the number of data corresponding to the bins here is more. After truncating and re-quantifying the output data set of this layer according to different truncation thresholds, a KL divergence distribution diagram corresponding to different truncation thresholds is obtained. See Figure 2b . Among them, Figure 2b the shown KL divergence curve diagram, the abscissa is the truncation threshold, and the ordinate is the KL divergence value.

[0067] According to Figure 2bIt can be seen that the KL curve is not smooth, and for concentrated distributions, there is a high probability of obtaining smaller quantization values. In principle, the distributions corresponding to adjacent truncation thresholds are extremely similar, and the KL divergence values corresponding to adjacent truncation thresholds should also be similar, rather than being as Figure 2b in the KL divergence curve in

[0068] where the KL divergence values corresponding to adjacent truncation thresholds vary greatly. For example, when the truncation thresholds are 128 and 129, the difference in KL divergence values is greater than 15%.

[0069] The embodiments of the present application provide a model quantization method, apparatus, electronic device, computer-readable storage medium, and computer program product based on optimized KL divergence, aiming to solve the above technical problems in the prior art. Among them, this method is applicable to terminal devices, which can be devices such as computers and servers. Specifically, for a model of the Float32 type, quantization of the Int8 type is performed to obtain a model based on the Int8 type. The embodiments of the present application also provide a KL divergence curve corresponding to the optimized KL divergence set (which can also be said to be an optimized KL divergence curve graph), and this curve is smoother and closer to the true KL divergence distribution compared to the KL divergence curve obtained by processing according to the original method.

[0070] See Figure 4 , a schematic flowchart of a model quantization method based on optimized KL divergence is provided in the present application. This method includes S410 - S430, where

[0071] S410, obtain the initial KL divergence set of any layer in the original model, where the initial KL divergence set includes initial KL divergence values corresponding one by one to each truncation threshold in the preset truncation threshold set, and any initial KL divergence value is determined by the following probability distributions: the original probability distribution of the output data set of this layer, the probability distribution of the new output data set determined by the corresponding truncation threshold, and the probability distribution of the new output data set is obtained according to the target quantization type.

[0072] S420, perform denoising processing on the initial KL divergence set according to the preset filtering algorithm to obtain the optimized KL divergence set corresponding to this layer.

[0073] S430, determine the output quantization value of this layer according to the target truncation threshold of this layer to perform quantization on this layer, where the target truncation threshold is the truncation threshold corresponding to the smallest KL divergence value in the optimized KL divergence set.

[0074] The method provided by the embodiments of the present application includes: first, obtaining the initial KL divergence set of any layer in the original model, and then denoising the initial KL divergence set according to a preset filtering algorithm to obtain the optimized KL divergence set corresponding to any layer. Since the optimized KL divergence set is obtained after denoising by the preset filtering algorithm, the KL divergence values corresponding to adjacent truncation thresholds do not differ much, which conforms to the principle that the distributions corresponding to adjacent truncation thresholds are extremely similar. On this basis, the target truncation threshold determined according to the optimized KL divergence set is more accurate. If the original model is quantized based on this method, the processing effect of the quantized model is not much different from that of the original model.

[0075] An embodiment of the present application provides a possible implementation manner: S410: Obtain the initial KL divergence set of any layer in the original model, which may specifically include S411-S412, where S412 includes S412a-S412d (the above steps are not shown in the attached drawings):

[0076] S411, input the calibration matrix into the original model for processing to obtain the output data set of each layer of the original model and the original probability distribution of the corresponding output data set;

[0077] Among them, the original model may be a model of float32 type and includes multiple layers. This model records the connection relationships between each layer, the processing algorithms of each layer, and shared weight information, etc. Each layer of the original model includes an input end and an output end. The input end is used to receive data to be processed, and the output end outputs the processing result of this layer. In the embodiments of the present application, multiple calibration matrices can be received through the input end of the first layer of the original model, and the set of all data output by the output end of each layer can be used as the output data set.

[0078] Among them, pictures are stored and participate in operations in the form of matrices in a computer. In the embodiments of the present application, the above-mentioned multiple calibration matrices are multiple calibration pictures. It should be noted that the calibration set pictures input into the original model are preprocessed, and the preprocessing process can refer to the prior art, and the present application does not limit this.

[0079] In a possible implementation manner, the process of obtaining the original probability distribution of the output data set of this layer includes:

[0080] After obtaining the output data set of this layer, screen the maximum absolute value of the output data set of this layer; determine the original probability distribution of the output data set of this layer according to the maximum absolute value.

[0081] Exemplarily, limited by the computing resources of the terminal device and the depth of the model, etc., generally two processes are adopted to obtain information such as the output dataset and the original probability distribution. First, multiple calibration matrices are input into the original model for processing to obtain the output dataset of each layer. Among them, calculate the maximum absolute value absmax of the output dataset of any layer (after the picture matrix is processed, negative numbers may appear in the obtained matrix. The absolute value of the negative number can be taken, and then the absolute value is included in the range of calculating the absolute value. When determining the output dataset later, the negative number is also replaced by its absolute value as a member of the output dataset); Second, multiple calibration matrices are input into the original model for processing again to obtain the output dataset of each layer. Among them, for the output dataset of any layer, divide 0 - absmax into N bins (in the prior art, N generally takes a value of 2048), and the width of each bin is: absmax / 2048. All the data in the output dataset are respectively counted onto the corresponding bins to obtain the histogram of the output dataset, and the probability distribution is further determined according to this histogram. This probability distribution is the original probability distribution corresponding to the output dataset.

[0082] S412, perform the following operations on the output dataset of any layer according to each truncation threshold in the preset truncation threshold set:

[0083] S412a, perform truncation processing on the output dataset of any layer according to the truncation threshold to obtain a new output dataset; perform quantization on the new output dataset according to the target quantization type, and determine the probability distribution of the new output dataset;

[0084] Among them, the target quantization type includes, but is not limited to, any one of the following types: any one of int2, int4, int8, int16.

[0085] Exemplarily, if the target quantization type is int8, it means that when re - segmenting and statistically analyzing the newly obtained output data set, a standard of 128 bins is used, and the truncation threshold set is 128 - 2048. In the above example, the output data set is divided into 2048 bins. Then, starting from the 128th bin, the output data set is truncated sequentially. Specifically, when the truncation threshold is 128, the data between the 1st bin and the 128th bin (corresponding to the above - mentioned new output data set) is statistically analyzed, and then the statistical result is re - segmented and statistically analyzed according to 128 bins, and the probability corresponding to each bin is statistically analyzed to obtain the probability distribution corresponding to the truncation threshold of 128; when the truncation threshold is 1000, the data between the 1st bin and the 1000th bin (corresponding to the above - mentioned new output data set) is statistically analyzed, and then the statistical data is re - segmented and statistically analyzed according to the standard of 128 bins, and the probability corresponding to each bin is statistically analyzed to obtain the probability distribution corresponding to the truncation threshold of 1000. After performing the above - mentioned similar processing procedures on the output data set based on each truncation threshold, the probability distribution corresponding to each truncation threshold is obtained.

[0086] It should be noted that since the normal value range of int8 is: - 128 - 127, but the range of int8 allowed by the terminal device in the embodiments of the present application is: - 127 - 127. After taking the absolute value, its range becomes 0 - 127, that is, there are a total of 128 numbers. The smallest truncation threshold of 128 in the embodiments of the present application and the value "128" in the quantization according to the standard of 128 bins are referenced to this total number 128.

[0087] S412c, according to the probability distribution of the new output data set and the original probability distribution corresponding to the truncation threshold, calculate the initial kl - divergence value corresponding to the truncation threshold.

[0088] Specifically, the above - mentioned formula 3 can be used to statistically analyze the kl - divergence value between the two distributions. It should be pointed out that the process of calculating the kl - divergence value through formula 3 can refer to the prior art and will not be elaborated here.

[0089] Among them, there are two ways to determine the original probability distribution corresponding to the truncation threshold. First, the original probability distribution of the output data set is used as the probability distribution corresponding to this truncation threshold. Second, after performing the truncation processing on the output data set, the data in the blocks (bins) after the truncation threshold is statistically analyzed into the last block, so as to obtain a new probability distribution, and this new probability distribution is determined as the original probability distribution corresponding to the truncation threshold.

[0090] S412d, determine all the initial kl - divergence values corresponding to the truncation thresholds as the initial kl - divergence set of this arbitrary layer.

[0091] Exemplarily refer to Figure 2b the KL divergence curve shown. After obtaining the KL divergence set of any layer, a KL divergence curve can be generated according to the KL divergence set of any layer. The difference between adjacent truncation thresholds in this curve is relatively large, and many KL divergence values are inaccurate.

[0092] An embodiment of the present application provides a possible implementation manner, S420: Denoise the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to any layer, which may specifically include:

[0093] Denoise the initial KL divergence set according to the one-dimensional Gaussian filtering algorithm to obtain an optimized KL divergence set.

[0094] Specifically, use the one-dimensional Gaussian filtering algorithm shown in the above formula 2 to filter the KL divergence set to obtain an optimized KL divergence set. Exemplarily refer to Figure 3 , after obtaining the optimized KL divergence set of any layer, a new KL divergence curve can be generated according to the optimized KL divergence set of any layer. It can be intuitively understood from this figure that the KL divergence values corresponding to adjacent truncation thresholds are not very different, which conforms to the principle that the distributions of adjacent truncation thresholds are extremely similar. Therefore, the optimized KL divergence set can be used as a reference for determining the target truncation threshold.

[0095] In a possible implementation manner, determine the output quantization value of any layer according to the target truncation threshold of any layer to quantize any layer, including: Determine the output quantization value of any layer according to the target truncation threshold and the width of the block.

[0096] Exemplarily, the target truncation threshold is m (m is between the truncation threshold 128 and the truncation threshold 2048), and the width of bins is the width of each bin after dividing the data in the range of 0 to absmax into 2048 bins. Therefore, the width of bins = absmax / 2048. The output quantization value out-scale can be calculated according to the following formula 4:

[0097] out-scale = (m + 0.5) * (absmax / 2048) / 127 Formula 4

[0098] It should be noted that 127 in Formula 4 is the maximum absolute value 127 in the range of the above-mentioned int8 (i.e., -127 to 127).

[0099] An embodiment of the present application also provides a possible implementation manner. Any layer includes shared weights, and the method further includes:

[0100] Determine the shared quantization value of the shared weights of any layer according to the target quantization type.

[0101] Exemplarily, the target quantization type is int8, and the shared weights exist in the form of a matrix in the original model. Since the shared weights are relatively fixed parameters in the original model and are rarely adjusted, the process of quantifying them can either refer to the above process of quantifying the output data set or calculate the mean value = absmax / 127 by statistically calculating the absmax of the shared weights, and use this mean value as the shared quantization value.

[0102] The embodiment of the present application also provides a possible implementation. After determining the output quantization value and the shared quantization value of any layer, the method further includes:

[0103] Determine the output quantization value and the shared quantization value of each layer in all layers except any layer in the original model; Quantify the original model according to the output quantization value and the shared quantization value of each layer in the original model layer, so as to obtain the quantified model.

[0104] In order to be able to determine the optimized performance of the quantified model, the performance of the quantified model can also be evaluated through the acc metric. Specifically, multiple calibration pictures are sequentially input into the original model and the quantified model for processing respectively, and the recognition accuracy of each calibration picture is calculated. Finally, the acc metric of the original model is 1, and the acc metric of the quantified model is 0.9. That is to say, compared with the original model, the performance of the quantified model only drops by 0.1. It should be noted that the process of how to obtain the acc metric can specifically refer to the prior art. In addition, other metrics can also be used to measure the performance difference between the quantified model and the original model, and the present application does not limit this.

[0105] See Figure 5 , the embodiment of the present application provides a structural schematic diagram of a model quantization device based on optimized kl divergence. The device 500 may include:

[0106] An acquisition module 510, configured to acquire an initial kl divergence set of any layer in the original model, where the initial kl divergence set includes initial kl divergence values corresponding one by one to each truncation threshold in the preset truncation threshold set, and any initial kl divergence value is determined by the following probability distributions: the original probability distribution of the output data set of any layer, the probability distribution of the new output data set determined by the corresponding truncation threshold, and the probability distribution of the new output data set is obtained according to the target quantization type;

[0107] A processing module 520, configured to perform denoising processing on the initial kl divergence set according to a preset filtering algorithm to obtain an optimized kl divergence set corresponding to any layer;

[0108] A determination module 530 is configured to determine the output quantization value of any layer according to the target truncation threshold of that layer, so as to perform quantization on that layer, where the target truncation threshold is the truncation threshold corresponding to the minimum KL divergence value in the optimized KL divergence set.

[0109] In a possible implementation, the obtaining module 510, when obtaining the initial KL divergence set of any layer in the original model, is specifically configured to:

[0110] Input the calibration matrix into the original model for processing to obtain the output data set of each layer of the original model and the original probability distribution of the corresponding output data set;

[0111] Perform the following operations on the output data set of any layer according to each truncation threshold in the preset truncation threshold set:

[0112] Truncate the output data set of any layer according to the truncation threshold to obtain a new output data set;

[0113] Quantize the new output data set according to the target quantization type and determine the probability distribution of the new output data set;

[0114] Calculate the initial KL divergence value corresponding to the truncation threshold according to the probability distribution of the new output data set and the original probability distribution corresponding to the truncation threshold;

[0115] Determine the initial KL divergence values corresponding to all truncation thresholds as the initial KL divergence set of any layer.

[0116] In a possible implementation, the obtaining module 510, in the process of obtaining the original probability distribution of the output data set of any layer, is specifically configured to:

[0117] After obtaining the output data set of any layer, filter out the maximum absolute value of the output data set of any layer;

[0118] Determine the original probability distribution of the output data set of any layer according to the maximum absolute value.

[0119] In a possible implementation, the processing module 520, when denoising the initial KL divergence set according to the preset filtering algorithm to obtain the optimized KL divergence set corresponding to any layer, is specifically configured to:

[0120] Denoise the initial KL divergence set according to the one-dimensional Gaussian filtering algorithm to obtain the optimized KL divergence set.

[0121] In a possible implementation, when any layer includes shared weights, the determination module 530 can also be used to:

[0122] Determine the shared quantization value of the shared weights of any layer according to the target quantization type.

[0123] In a possible implementation, after determining the output quantization value and the shared quantization value of any layer, the determining module 530 can also be used for:

[0124] Determine the output quantization value and the shared quantization value of each layer in all layers of the original model except the any layer;

[0125] Quantize the original model according to the output quantization value and the shared quantization value of each layer in the original model layer.

[0126] In an embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the model quantization method based on the optimized KL divergence shown in the present application.

[0127] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model quantization method based on the optimized KL divergence shown in the present application are implemented.

[0128] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the model quantization method based on the optimized KL divergence shown in the present application are implemented.

[0129] Terms such as "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification, claims and the above drawings of the present application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than the illustrated or textually described order.

[0130] It should be understood that although the flowchart in the embodiment of the present application indicates each operation step by an arrow, the execution order of these steps is not limited to the order indicated by the arrow. Unless there is a clear description in this article, in some implementation scenarios of the embodiment of the present application, the execution steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage among these sub-steps or stages can also be executed at different times respectively. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiment of the present application does not limit this.

[0131] The above are only alternative implementation manners of some implementation scenarios of this application. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of this application, adopting other similar implementation means based on the technical idea of this application also belongs to the protection scope of the embodiments of this application.

Claims

1. A model quantization method based on optimized KL divergence, characterized in that, The method includes: Obtaining an initial KL divergence set of any layer in the original model, where the original model is a deep neural network model, and the initial KL divergence set includes initial KL divergence values corresponding one by one to each truncation threshold in a preset truncation threshold set. Any initial KL divergence value is determined by the following probability distributions: the original probability distribution of the output data set of the any layer, and the probability distribution of a new output data set determined by the corresponding truncation threshold. The probability distribution of the new output data set is obtained according to the target quantization type; Performing denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to the any layer; Determining the output quantization value of the any layer according to the target truncation threshold of the any layer for quantization, where the target truncation threshold is the truncation threshold corresponding to the smallest KL divergence value in the optimized KL divergence set; Wherein, the obtaining of the initial KL divergence set of any layer in the original model includes: Inputting a calibration matrix into the original model for processing to obtain the output data set of each layer of the original model and the original probability distribution of the corresponding output data set; the calibration matrix includes a picture matrix; Performing truncation processing on the output data set of the any layer according to the truncation threshold to obtain a new output data set; Quantizing the new output data set according to the target quantization type and determining the probability distribution of the new output data set; Calculating the initial KL divergence value corresponding to the truncation threshold according to the probability distribution of the new output data set and the original probability distribution corresponding to the truncation threshold; Determining the initial KL divergence values corresponding to all truncation thresholds as the initial KL divergence set of the any layer.

2. The method according to claim 1, characterized in that The process of obtaining the original probability distribution of the output data set of the any layer includes: After obtaining the output data set of the any layer, screening the maximum absolute value of the output data set of the any layer; Determining the original probability distribution of the output data set of the any layer according to the maximum absolute value.

3. The method according to claim 1, characterized in that The performing of denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to the any layer includes: Performing denoising processing on the initial KL divergence set according to a one-dimensional Gaussian filtering algorithm to obtain the optimized KL divergence set.

4. The method according to any one of claims 1 to 3, characterized in that The any layer includes shared weights, and the method further includes: Determining the shared quantization value of the shared weights of the any layer according to the target quantization type.

5. The method according to claim 4, characterized in that After determining the output quantization value and the shared quantization value of the any layer, the method further includes: Determining the output quantization value and the shared quantization value of each layer in all layers except the any layer in the original model; Quantizing the original model according to the output quantization value and the shared quantization value of each layer in the original model layer.

6. An model quantization device based on optimized KL divergence, characterized in that, The device includes: An acquisition module, configured to acquire an initial KL divergence set of any layer in an original model, where the original model is a deep neural network model, the initial KL divergence set includes initial KL divergence values corresponding one by one to each truncation threshold in a preset truncation threshold set, and any initial KL divergence value is determined by the following probability distributions: the original probability distribution of the output data set of the any layer, and the probability distribution of a new output data set determined by a corresponding truncation threshold, and the probability distribution of the new output data set is obtained according to a target quantization type; A processing module, configured to perform denoising processing on the initial KL divergence set according to a preset filtering algorithm to obtain an optimized KL divergence set corresponding to the any layer; A determination module, configured to determine an output quantization value of the any layer according to a target truncation threshold of the any layer to perform quantization on the any layer, where the target truncation threshold is the truncation threshold corresponding to the smallest KL divergence value in the optimized KL divergence set; Wherein, when the acquisition module acquires the initial KL divergence set of any layer in the original model, it is specifically configured to: Input a calibration matrix into the original model for processing to obtain an output data set of each layer of the original model and the original probability distribution of the corresponding output data set; the calibration matrix includes a picture matrix; Perform truncation processing on the output data set of the any layer according to the truncation threshold to obtain a new output data set; Quantize the new output data set according to the target quantization type and determine the probability distribution of the new output data set; Calculate the initial KL divergence value corresponding to the truncation threshold according to the probability distribution of the new output data set and the original probability distribution corresponding to the truncation threshold; Determine the initial KL divergence values corresponding to all truncation thresholds as the initial KL divergence set of the any layer.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-5 are implemented.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-5 are implemented.

Images