Model quantification method and device, related equipment and computer program product

Abstract

The invention discloses a model quantification method and device, related equipment and a computer program product, and aims to realize a better quantification strategy by taking each layer of a target model as the quantification granularity instead of taking the whole target model as the quantification granularity in view of possible different sensitivities of different levels in the model to the quantification. And determining a quantization target of each layer according to the weight and the activation value of each layer, and further performing quantization processing on each layer in the target model according to the quantization target of each layer. According to the method, the quantization target matched with each layer of the target model is determined, so that each layer can be quantized according to the more matched quantization target, and compared with a traditional scheme that the whole model is processed by adopting the same quantization target, the quantization strategy of the scheme is more excellent, and the quantization efficiency is improved. The parameter quantity and the calculation precision of the quantized model can be better balanced, and the model quantization effect is improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and more specifically, to a model quantization method, apparatus, related equipment and computer program product. Background Art

[0002] In the field of deep learning model optimization, quantization is a common technique that aims to reduce the size of the model and accelerate the inference process by reducing the data representation accuracy of the model parameters. Taking the Transformer model as an example, in recent years, this model has achieved remarkable success in fields such as natural language processing, but they have a huge number of parameters and high demand for computing resources. Therefore, developing a tool that can automatically quantize is crucial to improving the usability of these models in resource-constrained environments.

[0003] In the prior art, there have been some attempts to quantize deep learning models, such as the static quantization scheme. This quantization scheme is applied after the model training is completed. The quantization target for the entire model is set manually based on experience and experiments, and then all weights and activations in the entire model are quantized according to the quantization target. The quantization strategy of this quantization scheme is not good enough and still needs to be improved. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a model quantization method, apparatus, related equipment and computer program product to improve the excellence of the model quantization strategy. The specific scheme is as follows:

[0005] In a first aspect, a model quantization method is provided, comprising:

[0006] Analyze the structure of the target model to be quantized, and obtain the weights and activations of each layer in the target model;

[0007] Taking each layer of the target model as the quantization granularity, determining the quantization target of each layer according to the weight and activation values ​​of each layer, wherein the quantization target includes the quantization targets of the weight and activation respectively;

[0008] According to the quantization target of each layer, each layer of the target model is quantized respectively.

[0009] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the quantization target of each layer according to the weight and activation value of each layer includes:

[0010] The weights and activations of each layer are jointly optimized according to the following formula to obtain the optimized weights and optimized activations:

[0011]

[0012] Among them, W and X represent the weight and activation before optimization, respectively. and They represent the optimized weight and the optimized activation respectively, and diag(s) is a function of s0 to s n The diagonal matrix is ​​arranged, j takes values ​​from 1 to n, n is the total number of activated input channels, max(|W j |) represents the maximum weight of the jth channel in the current layer of the target model, max(|X j |) represents the maximum activation value of the jth channel of the current layer obtained by the target model in the forward reasoning process of the samples in the sample set, and α is a hyperparameter;

[0013] For each layer of the target model:

[0014] Traverse each candidate hyperparameter value in the candidate hyperparameter set and calculate the value of α for each candidate hyperparameter value and The candidate hyperparameter set includes more than two candidate hyperparameter values, and each candidate hyperparameter value is a value greater than 0 and less than 1;

[0015] For each value of a, we can calculate Select the smallest The corresponding a value is taken as the target value of the current layer α, and according to the minimum Determines the quantization target for the current layer.

[0016] In a possible design, in another implementation of the first aspect of the embodiment of the present application, according to the minimum The process of determining the quantitative target of the current layer includes:

[0017] If the smallest If the activation bit width of the current layer is greater than the first set threshold, the bit width before and after the activation bit width of the current layer is kept unchanged, and the quantization target of the weight of the current layer is set to a lower bit width than before the quantization bit width;

[0018] If the smallest If it is not greater than the first set threshold, the quantization target of the weight after optimization of the current layer is set to a lower bit width than before quantization, and the quantization target of the activation after optimization of the current layer is set to a lower bit width than before quantization.

[0019] In a possible design, in another implementation of the first aspect of the embodiment of the present application, in the minimum Under the condition that the value is greater than the first set threshold, the process of setting the quantization target of the weight of the current layer to a lower bit width than before quantization includes:

[0020] In the smallest When the value is greater than the first threshold value:

[0021] If max(W) is less than or equal to the second set threshold, the quantization target of the weight of the current layer is set to W2, where W represents the weight and 2 represents the bit width after quantization;

[0022] If max(W) is greater than the second set threshold and less than or equal to the third set threshold, the quantization target of the weight of the current layer is set to W4, where 4 represents the bit width after quantization;

[0023] If max(W) is greater than the third set threshold and less than or equal to the fourth set threshold, the quantization target of the weight of the current layer is set to W8, where 8 represents the bit width after quantization;

[0024] The second set threshold is smaller than the third set threshold, and the third set threshold is smaller than the fourth set threshold.

[0025] In a possible design, in another implementation of the first aspect of the embodiment of the present application, in the minimum The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the quantization target of the weight after optimization of the current layer is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, including:

[0026] In the smallest Under the condition that it is not greater than the first set threshold:

[0027] If the smallest If the value is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A4, where W represents weight, A represents activation, and the number 4 represents the bit width after quantization, and the fifth set threshold is less than the first set threshold.

[0028] In a possible design, in another implementation of the first aspect of the embodiment of the present application, in the minimum The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the weight is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, further includes:

[0029] In the smallest Under the condition that it is not greater than the first set threshold:

[0030] If the smallest If it is greater than the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W8, and the quantization target of the activation after optimization of the current layer is set to A8;

[0031] like is greater than the fifth set threshold and less than or equal to the first set threshold, and, If it is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A8.

[0032] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of performing quantization processing on each layer of the target model according to the quantization target of each layer includes:

[0033] For each layer of the target model:

[0034] If the quantization target of the current layer indicates to keep activation without quantization and only quantize weights, the weights of the current layer are quantized offline according to the set quantization algorithm;

[0035] If the quantization target of the current layer indicates that both the optimized weights and the optimized activations of the current layer are quantized, the optimized weights of the current layer are quantized offline according to the set quantization algorithm. At the same time, diag(s) is saved for the online inference process of the target model. Reasoning is performed on the real-time input data and the activation of the current layer is obtained. The activation of the current layer is divided by diag(s) as the optimized activation, and the optimized activation of the current layer is quantized online according to the set quantization algorithm.

[0036] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the setting quantization algorithm adopts the following formula:

[0037]

[0038] Among them, V i，j Indicates the elements in the weight or activation matrix that need to be quantized, Quant(V i，j ) represents the quantized result, and m represents the quantization bit width;

[0039] Based on this, if the quantization target of the current layer means that both the optimized weights and the optimized activations of the current layer are quantized, when saving diag(s), max(|X j |) for online quantization of the current layer after optimization, and save the max(|X j |) as the max|V in the quantization algorithm j |.

[0040] In a second aspect, a model quantization device is provided, comprising:

[0041] A model parsing unit, used to parse the structure of the target model to be quantized, and obtain the weight and activation of each layer in the target model;

[0042] A quantization target determination unit, configured to determine a quantization target for each layer of the target model according to the weight and activation values ​​of each layer, using each layer as a quantization granularity, wherein the quantization target includes quantization targets for each weight and activation;

[0043] The quantization processing unit is used to perform quantization processing on each layer of the target model according to the quantization target of each layer.

[0044] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0045] The memory is used to store programs;

[0046] The processor is used to execute the program to implement the model quantization method described in any one of the first aspects of the present application.

[0047] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the model quantization method described in any one of the first aspects of the present application is implemented.

[0048] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the model quantization method described in any one of the first aspects of the present application.

[0049] By means of the above technical solution, the present application analyzes the target model to be quantized, and obtains the weights and activations of each layer therein, so as to facilitate the subsequent quantization of the weights and activations. Furthermore, the present application takes into account that the sensitivity of different levels within the model to quantization may be different. In order to achieve a better quantization strategy, the present application does not use the entire target model as the quantization granularity, but uses each layer of the target model as the quantization granularity. According to the numerical value of the weight and activation of each layer, the quantization target of each layer is determined. The quantization target includes the quantization target of the weight and activation of the current layer, and then each layer in the target model can be quantized according to the quantization target of each layer. Obviously, the present application provides a more fine-grained quantization method, and a matching quantization target is determined for each layer of the target model, so that each layer can be quantized according to a more matching quantization target. Compared with the traditional solution that uses the same quantization target to process the entire model, the quantization strategy of the present application solution is more excellent, which can better balance the parameter quantity and calculation accuracy of the quantized model and improve the effect of model quantization. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0051] Figure 1 A schematic diagram of a process flow of a model quantization method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of the structure of a model quantization device provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Existing model quantization solutions often require manual design of quantization strategies based on the specific characteristics of the model and application scenarios. These solutions may have the following disadvantages:

[0056] 1. Treat the model as a whole: If quantization is enabled, the entire quantization operation is applied to the entire model, that is, the entire model uses the same quantization target. This solution rarely takes into account that different levels within the model may have different sensitivities to quantization, thus failing to achieve the optimal quantization strategy.

[0057] 2. Manual adjustment is time-consuming: Finding the right quantitative target usually requires repeated trials, which is a time-consuming process and relies on manual expertise.

[0058] 3. Limited generalization ability: A specific quantitative target may only be applicable to a certain type of model or task, and needs to be redesigned for other models or tasks.

[0059] In response to the above problems, this application aims to provide an automated layer-by-layer quantization solution that can adaptively generate quantization targets based on model characteristics, reduce manual intervention, improve quantization efficiency, and have strong generalization capabilities.

[0060] The model quantization method provided in this application can be applied to quantize various types of models, including but not limited to Transformer models, or other models that contain many parameters and need to be quantized to improve reasoning efficiency and are deployed on terminal devices. For ease of description, this application uses the Transformer model as an example of the target model to be quantized.

[0061] The model quantization method provided in this application can realize offline quantization processing, that is, the model quantization method of this application is realized after the target model is trained and before the online reasoning task is performed. Based on this, the model quantization method of this application can be applied to offline devices, such as terminals, servers, etc.

[0062] In addition, the model quantization method provided in the present application can also be a part of offline quantization processing and another part of online quantization processing. For example, for the weights in the target model, the quantization processing can be performed offline, and for the activations in the target model, the quantization processing can be performed online. Based on this, the model quantization method of the present application can be applied to a system composed of offline devices and online devices. Among them, the online device can be a device deployed with the target model, such as a mobile phone, computer, car computer, learning machine and other terminals, and the offline device can be a server, cloud, etc.

[0063] The "online" referred to in this application refers to the scenario where the target model performs reasoning tasks, and correspondingly, the "online device" refers to the device that is deployed with the target model to perform reasoning tasks using the target model. The "offline" referred to in this application is relative to the "online", and the "offline" refers to the scenario where the target model does not perform reasoning tasks, which can generally be understood as the scenario before the target model performs reasoning tasks. Correspondingly, the "offline device" is a device other than the "online device", that is, the device where the target model is located before it is deployed to the "online device", such as a server, cloud, etc.

[0064] Next, a model quantization method provided in an embodiment of the present application is introduced, and the method is applied to a computer device as an example. The computer device can be an offline device or a system composed of an offline device and an online device. Figure 1 , the model quantization method specifically includes the following steps:

[0065] Step S100: parse the structure of the target model to be quantized to obtain the weight and activation of each layer in the target model.

[0066] Specifically, the target model is generally composed of a multi-level structure, each level includes a weight matrix and an activation matrix. During model inference, matrix multiplication operations account for the vast majority of the overall calculation and are the main target of quantization. In order to facilitate the subsequent determination of quantization targets, in this step, by parsing the target model, the weight W and activation X of each level in the target model can be obtained.

[0067] Step S110, taking each layer of the target model as the quantization granularity, and determining the quantization target of each layer according to the numerical values ​​of the weight and activation of each layer, wherein the quantization target includes the quantization targets for the weight and activation respectively.

[0068] Specifically, in this embodiment, taking into account that different layers of the model may have different sensitivities to quantization, in order to achieve a better quantization strategy, in this step, a single layer in the target model is used as the quantization granularity, and a quantization target that matches each layer is determined.

[0069] Specifically, for each layer, the quantization target of the current layer is automatically determined according to the numerical values ​​of the weight and activation of the current layer, and the quantization target includes the quantization targets for the weight and activation respectively.

[0070] Among them, the quantization target can be understood as the specific effect and purpose expected to be achieved by the quantization process. Exemplarily, the quantization targets supported by the present application include but are not limited to: W4A4, W8A8, W8A16, etc. Among them, W represents weight, A represents activation, and the number represents the bit width after quantization, such as W4A4 means that both weight and activation are quantized to 4 bits.

[0071] It should be noted that if the bit width after quantization is the same as the bit width before quantization, it can be understood that no quantization is performed. For example, if the original activation of the model is a floating point number FP16, then A16 means that the bit width after quantization is still 16, that is, the bit width before and after quantization remains unchanged, which can be understood as keeping the activation unquantized.

[0072] The process of determining the quantization target can comprehensively consider the accuracy loss of the quantized model and the size of the parameters of the quantized model, and balance the relationship between the two to select the appropriate quantization target.

[0073] Considering that the weights are solidified after the target model training is completed, the quantization target of the weights can be directly obtained offline. Activation is a value that can only be determined by an online reasoning process. Therefore, the process of determining the quantization target of activation in this step can be determined by an online reasoning process. Of course, in order to further improve the reasoning efficiency, this step can also be performed offline, using a small amount of configured sample data to send to the target model for reasoning, obtaining activation in the process, and estimating the quantization target of activation based on the activation calculated by the sample reasoning.

[0074] Step S120: quantize each layer of the target model according to the quantization target of each layer.

[0075] Specifically, after determining the quantization target of each layer in the target model, each layer can be quantized according to the corresponding quantization target. The quantization process can be performed in an online inference state, or the weights can be quantized in an offline state and the activations can be quantized in an online inference state.

[0076] The model quantization method provided in the embodiment of the present application analyzes the target model to be quantized to obtain the weights and activations of each layer, so as to facilitate the subsequent quantization of the weights and activations. Furthermore, the present application takes into account that the sensitivity of different levels within the model to quantization may be different. In order to achieve a better quantization strategy, the present application does not use the entire target model as the quantization granularity, but uses each layer of the target model as the quantization granularity. According to the numerical value of the weight and activation of each layer, the quantization target of each layer is determined. The quantization target includes the weight of the current layer and the quantization target of each activation. Then, each layer in the target model can be quantized according to the quantization target of each layer. Obviously, the present application provides a more fine-grained quantization method, and a matching quantization target is determined for each layer of the target model, so that each layer can be quantized according to a more matching quantization target. Compared with the traditional solution that uses the same quantization target to process the entire model, the quantization strategy of the present application solution is more excellent, which can better balance the parameter quantity and calculation accuracy of the quantized model and improve the effect of model quantization.

[0077] In some embodiments of the present application, considering that after the target model training is completed, its fixed weight value is generally small and evenly distributed, which is very friendly to quantization. However, the activation value often has a large dynamic range and a large quantization error. In order to reduce the activation quantization error, in this embodiment, before quantizing the weights and activations of each layer, an operation of jointly optimizing the weights and activations of each layer can be added to obtain the optimized weights and optimized activations. An example of the joint optimization operation is as follows:

[0078]

[0079] Among them, W and X represent the weight and activation before optimization, respectively. and They represent the optimized weight and the optimized activation respectively, and diag(s) is a function of s0 to s nThe diagonal matrix is ​​arranged, j takes values ​​of 1 to n, n is the total number of activated input channels, and α is a hyperparameter, which can be understood as the migration strength, which controls how much of the quantization difficulty of the activation value is migrated to the weight quantization. Balance the quantization difficulty between activation and weight, so that each can be easily quantized. A suitable migration strength value α can make both weights and activations easy to quantize. In this embodiment, α is set to a dynamic value, that is, for different layers in the target model, α dynamically selects the optimal value.

[0080] For max(|X j |) and max(|W j |):

[0081] max(|W j |) represents the maximum weight of the jth channel in the current layer of the target model. Since the weight W is solidified after the target model training is completed, max(|W) can be directly obtained offline. j |).

[0082] The activation X is a value that can only be determined by online reasoning. In order to be able to determine max(|X j |), thereby improving the efficiency of online reasoning of the model. In this embodiment, a small number of samples can be used for estimation in an offline state.

[0083] Specifically, the user can pre-configure the sample set path, and then the method of the present application automatically extracts a specified number of samples from the sample set and inputs them into the target model for forward reasoning, obtains the intermediate activation value, and saves the maximum value of each activated channel as offline data, and then can obtain max(|X j |). That is, in this embodiment, max(|X j |) can be expressed as the maximum activation value of the jth channel of the current layer obtained during the forward reasoning process of the target model on the samples in the sample set.

[0084] The process of determining quantitative goals:

[0085] For each layer of the target model:

[0086] In order to select the most appropriate hyperparameter α for the current layer, the present application may pre-configure a candidate hyperparameter set, which includes more than two candidate hyperparameter values, and each candidate hyperparameter value is a value greater than 0 and less than 1. Exemplarily, the candidate hyperparameter values ​​included in the candidate hyperparameter set include, but are not limited to, values ​​such as 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9. In the following embodiments, for ease of description, the candidate hyperparameter set including the above 9 values ​​is taken as an example for illustration.

[0087] This application can traverse each candidate hyperparameter value in the candidate hyperparameter set, and calculate the value of α when each candidate hyperparameter value is taken. and

[0088] For each value of a, calculate Then when a has 9 candidate values, we can get 9 In this embodiment, the smallest The corresponding a value is taken as the target value of the current layer α, and according to the minimum Determines the quantization target for the current layer.

[0089] This application can pre-configure several candidate quantization targets, and then can be based on the above minimum To determine the quantitative target of the current layer matching.

[0090] Low-bit quantization essentially uses a quantization algorithm to map data represented by high bits (such as 32 bits, 16 bits, etc.) to low bits (8 bits, 4 bits, 2 bits, etc.). Map a larger range of data to a smaller range. If the maximum value of the weight or activation channel is larger, the channel value range is larger, and the error is greater when the quantization is compressed to a lower bit data representation. Therefore, by comprehensively considering the precision loss after quantization and the size of the parameters after quantization, the above minimum Determine quantitative goals.

[0091] This embodiment provides a method based on the above minimum Determine the options for achieving the quantitative goals. Specifically:

[0092] Case 1: If the smallest If the value of quantization_weight is greater than the first set threshold (defined as v), the bit width of the activation of the current layer before and after quantization is kept unchanged (that is, the activation is not quantized), and the quantization target of the weight of the current layer (it should be noted that the weight here refers to the original weight, not the optimized weight) is set to a lower bit width than before quantization.

[0093] Case 2: If the smallest If it is not greater than the first set threshold value v, the quantization target of the weight after optimization of the current layer is set to a lower bit width than before quantization, and the quantization target of the activation after optimization of the current layer is set to a lower bit width than before quantization.

[0094] That is, in the above minimum When it is not greater than v, it indicates that the joint optimization of the above weights and activations is beneficial. Therefore, the subsequent quantization objects are also quantized on the optimized weights and optimized activations.

[0095] The value of the first set threshold value v can be configured by the user, and the user can set the value of v according to factors such as the type of the target model.

[0096] Next, several optional implementations of the above situation 1 and situation 2 are introduced respectively.

[0097] It should be noted that, in the following implementation, the bit width of the original weight and activation before quantization is 16 bits or other bit widths greater than 16 bits as an example for illustration.

[0098] For case 1:

[0099] In the smallest When it is greater than the first set threshold v, it means that the maximum values ​​of the optimized activation and the optimized weight are still too large, which is not suitable for the joint optimization of the weight and activation in the previous step. Therefore, the quantization of the activation can be abandoned and only the original weight can be quantized.

[0100] if:

[0101] max(W)≤γ (3)

[0102] Then, the quantization target of the weight of the current layer is set to W2, and the activation remains unchanged. W represents the weight, and 2 represents the bit width after quantization. Among them, γ is the second set threshold.

[0103] if:

[0104] γ<max(W)≤δ (4)

[0105] Then, the quantization target of the weight of the current layer is set to W4, and the activation remains unchanged. W represents the weight, and 4 represents the bit width after quantization. Among them, δ is the third set threshold.

[0106] if:

[0107] δ<max(W)≤ε (5)

[0108] Then, the quantization target of the weight of the current layer is set to W8, and the activation remains unchanged. W represents the weight, and 8 represents the bit width after quantization. Among them, ε is the fourth set threshold.

[0109] The magnitude relationship between the above thresholds satisfies: γ<δ<ε.

[0110] For case 2:

[0111] In the smallest Under the condition that it is not greater than the first set threshold, the weight of the current layer is optimized and after optimization activation All are quantized to a lower bit width.

[0112] For the sake of convenience, the following will be the smallest Abbreviation

[0113] if:

[0114]

[0115] Then, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A4, where W represents weight, A represents activation, and the number 4 represents the bit width after quantization. Where u is the fifth set threshold, and u <v。

[0116] if:

[0117]

[0118] Then, the quantization target of the weight after optimization of the current layer is set to W8, and the quantization target of the activation after optimization of the current layer is set to A8, where W represents weight, A represents activation, and the number 8 represents the bit width after quantization.

[0119] if:

[0120]

[0121] Then, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A8.

[0122] The above embodiments of the present application provide several possible implementation methods for determining the quantization target of each layer. It is understandable that the above examples only illustrate several quantization targets supported by the scheme of the present application, such as W4A4, W4A8, W8A8, etc. The embodiments of the present application do not exhaustively list all the quantization targets supported by the scheme. Other quantization targets and selection methods expanded on the basis of the selection ideas of the quantization targets in the above examples of the present application also fall within the scope of protection of the present application.

[0123] After determining the quantization target of each layer in the target model, each layer of the target model can be quantized. This embodiment provides an optional implementation method of quantization, which specifically includes the following steps:

[0124] For each layer of the target model:

[0125] If the quantization target of the current layer indicates to keep the activation without quantization and only quantize the weight, the weight of the current layer is quantized offline according to the set quantization algorithm.

[0126] If the quantization target of the current layer indicates that both the optimized weights and optimized activations of the current layer are quantized, the optimized weights of the current layer are quantized offline according to the set quantization algorithm, and diag(s) is saved.

[0127] It is understandable that activation is a value that can only be determined in an online reasoning process, so it is impossible to accurately quantify activation in an offline state. The present application can save diag(s) calculated in the above formula (1) in an offline state, so that in the subsequent online reasoning process of the target model, reasoning is performed on the real-time input data and the activation of the current layer is obtained, and the activation of the current layer is divided by diag(s) as the optimized activation, and the optimized activation of the current layer is quantized online according to the set quantization algorithm.

[0128] The quantization algorithm mentioned in the above embodiment can adopt various types of quantization algorithms. In this embodiment, the RTN (round-to-nearest) quantization algorithm is used as an example for explanation. The quantization algorithm formula is expressed as follows:

[0129]

[0130] Among them, V i，j Indicates the elements in the weight or activation matrix that need to be quantized, Quant(V i，j ) represents the quantized result, and m represents the quantization bit width.

[0131] Take the above RTN quantization algorithm as an example:

[0132] If the quantization target of the current layer indicates that the activation is kept without quantization and only the weight is quantized, then the original weight of the current layer is quantized offline according to the above formula (10).

[0133] If the quantization target of the current layer indicates that both the optimized weights and the optimized activations of the current layer are quantized, then according to the above formula (10), the optimized weights of the current layer are offline quantized, and diag(s) is saved.

[0134] In the subsequent online inference process of the target model, inference is performed on the real-time input data to obtain the activation of the current layer. The activation of the current layer is divided by diag(s) as the optimized activation, which is substituted into the above formula (10) to quantize the optimized activation.

[0135] Further optionally, in order to further reduce the efficiency of online inference of the target model and reduce the amount of calculation, for max|V in the above formula (10), j |, the max(|X j|) is replaced. That is, if the quantization target of the current layer indicates that both the optimized weights and the optimized activations of the current layer are quantized, then according to the above formula (10), the optimized weights of the current layer are quantized offline, and at the same time, diag(s) and max(|X j |) for online quantization of the current layer after optimization, and save the max(|X j |) as the max|V in the quantization algorithm formula (10) j |, eliminating the process of online temporary calculations and improving the efficiency of online model reasoning.

[0136] In addition, in the above steps, while quantizing each layer of the target model according to the quantization target of each layer, the quantization mark can be further saved. The quantization mark corresponds to the quantization target one by one, which facilitates the subsequent target model inference stage to understand the quantization target used by each layer.

[0137] The model quantization device provided in an embodiment of the present application is described below. The model quantization device described below and the model quantization method described above can be referenced to each other.

[0138] See also Figure 2 , Figure 2 A schematic diagram of the structure of a model quantization device disclosed in an embodiment of the present application.

[0139] like Figure 2 As shown, the device may include:

[0140] A model parsing unit 11 is used to parse the structure of the target model to be quantized and obtain the weight and activation of each layer in the target model;

[0141] A quantization target determination unit 12, configured to use each layer of the target model as a quantization granularity and determine a quantization target for each layer according to the weight and activation values ​​of each layer, wherein the quantization target includes quantization targets for each weight and activation;

[0142] The quantization processing unit 13 is used to perform quantization processing on each layer of the target model according to the quantization target of each layer.

[0143] In a possible implementation, the process of determining the quantization target of each layer by the quantization target determination unit according to the weight and activation value of each layer includes:

[0144] The weights and activations of each layer are jointly optimized according to the following formula to obtain the optimized weights and optimized activations:

[0145]

[0146] Among them, W and X represent the weight and activation before optimization, respectively. and They represent the optimized weight and the optimized activation respectively, and diag(s) is a function of s0 to s n The diagonal matrix is ​​arranged, j takes values ​​from 1 to n, n is the total number of activated input channels, max(|W j |) represents the maximum weight of the jth channel in the current layer of the target model, max(|X j |) represents the maximum activation value of the jth channel of the current layer obtained by the target model in the forward reasoning process of the samples in the sample set, and α is a hyperparameter;

[0147] For each layer of the target model:

[0148] Traverse each candidate hyperparameter value in the candidate hyperparameter set and calculate the value of α for each candidate hyperparameter value and The candidate hyperparameter set includes more than two candidate hyperparameter values, and each candidate hyperparameter value is a value greater than 0 and less than 1;

[0149] For each value of a, we can calculate Select the smallest The corresponding a value is taken as the target value of the current layer α, and according to the minimum Determines the quantization target for the current layer.

[0150] In a possible implementation, the quantization target determination unit determines the minimum The process of determining the quantitative target of the current layer includes:

[0151] If the smallest If the activation bit width of the current layer is greater than the first set threshold, the bit width before and after the activation bit width of the current layer is kept unchanged, and the quantization target of the weight of the current layer is set to a lower bit width than before the quantization bit width;

[0152] If the smallest If it is not greater than the first set threshold, the quantization target of the weight after optimization of the current layer is set to a lower bit width than before quantization, and the quantization target of the activation after optimization of the current layer is set to a lower bit width than before quantization.

[0153] In a possible implementation, the quantization target determination unit determines the minimum Under the condition that the value is greater than the first set threshold, the process of setting the quantization target of the weight of the current layer to a lower bit width than before quantization includes:

[0154] In the smallest When the value is greater than the first threshold value:

[0155] If max(W) is less than or equal to the second set threshold, the quantization target of the weight of the current layer is set to W2, where W represents the weight and 2 represents the bit width after quantization;

[0156] If max(W) is greater than the second set threshold and less than or equal to the third set threshold, the quantization target of the weight of the current layer is set to W4, where 4 represents the bit width after quantization;

[0157] If max(W) is greater than the third set threshold and less than or equal to the fourth set threshold, the quantization target of the weight of the current layer is set to W8, where 8 represents the bit width after quantization;

[0158] Among them, the second set threshold is smaller than the third set threshold, the third set threshold is smaller than the fourth set threshold, and the fourth set threshold is smaller than the first set threshold.

[0159] In a possible implementation, the quantization target determination unit determines the minimum The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the quantization target of the weight after optimization of the current layer is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, including:

[0160] In the smallest Under the condition that it is not greater than the first set threshold:

[0161] If the smallest If the value is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A4, where W represents weight, A represents activation, and the number 4 represents the bit width after quantization, and the fifth set threshold is less than the first set threshold.

[0162] In a possible implementation, the quantization target determination unit determines the minimum The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the weight is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, further includes:

[0163] In the smallest Under the condition that it is not greater than the first set threshold:

[0164] If the smallest If it is greater than the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W8, and the quantization target of the activation after optimization of the current layer is set to A8;

[0165] like is greater than the fifth set threshold and less than or equal to the first set threshold, and If it is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A8.

[0166] In a possible implementation, the process of the quantization processing unit performing quantization processing on each layer of the target model according to the quantization target of each layer includes:

[0167] For each layer of the target model:

[0168] If the quantization target of the current layer indicates to keep activation without quantization and only quantize weights, the weights of the current layer are quantized offline according to the set quantization algorithm;

[0169] If the quantization target of the current layer indicates that both the optimized weights and the optimized activations of the current layer are quantized, the optimized weights of the current layer are quantized offline according to the set quantization algorithm. At the same time, diag(s) is saved for the online inference process of the target model. Reasoning is performed on the real-time input data and the activation of the current layer is obtained. The activation of the current layer is divided by diag(s) as the optimized activation, and the optimized activation of the current layer is quantized online according to the set quantization algorithm.

[0170] In a possible implementation, the setting quantization algorithm adopts the following formula:

[0171]

[0172] Among them, V i，j Indicates the elements in the weight or activation matrix that need to be quantized, Quant(V i，j ) represents the quantized result, and m represents the quantization bit width;

[0173] Based on this, when the quantization processing unit determines that the quantization target of the current layer represents the quantization of both the optimized weights and the optimized activations of the current layer, it saves diag(s) and also saves max(|X j |) for online quantization of the current layer after optimization, and save the max(|X j |) as the max|V in the quantization algorithm j |.

[0174] The present application also provides an electronic device in an embodiment. Figure 3 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, teaching large screens, wearable devices, etc. Figure 3The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0175] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303 to implement the model quantization method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in RAM 303. Processing device 301, ROM 302, and RAM 303 are connected to each other via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0176] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a memory card, a hard disk, etc.; and communication devices 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0177] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any model quantization method provided in the embodiment of the present application.

[0178] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any model quantization method provided in the embodiment of the present application.

[0179] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0180] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0181] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A model quantization method, characterized in that: include: Analyze the structure of the target model to be quantized, and obtain the weights and activations of each layer in the target model; Taking each layer of the target model as the quantization granularity, determining the quantization target of each layer according to the weight and activation values ​​of each layer, wherein the quantization target includes the quantization targets of the weight and activation respectively; According to the quantization target of each layer, each layer of the target model is quantized respectively.

2. The method according to claim 1, characterized in that The process of determining the quantization target for each layer based on the weights and activation values ​​of each layer includes: The weights and activations of each layer are jointly optimized according to the following formula to obtain the optimized weights and optimized activations: Among them, W and X represent the weight and activation before optimization, respectively. and They represent the optimized weight and the optimized activation respectively, and diag(s) is a function of s0 to s n The diagonal matrix is ​​arranged, j takes values ​​from 1 to n, n is the total number of activated input channels, max(|W j |) represents the maximum weight of the jth channel in the current layer of the target model, max(|X j |) represents the maximum activation value of the jth channel of the current layer obtained by the target model in the forward reasoning process of the samples in the sample set, and α is a hyperparameter; For each layer of the target model: Traverse each candidate hyperparameter value in the candidate hyperparameter set and calculate the value of α for each candidate hyperparameter value and The candidate hyperparameter set includes more than two candidate hyperparameter values, and each candidate hyperparameter value is a value greater than 0 and less than 1; For each value of a, we can calculate Select the smallest The corresponding a value is taken as the target value of the current layer α, and according to the minimum Determines the quantization target for the current layer.

3. The method according to claim 2, characterized in that According to the minimum The process of determining the quantitative target of the current layer includes: If the smallest If the activation bit width of the current layer is greater than the first set threshold, the bit width before and after the activation bit width of the current layer is kept unchanged, and the quantization target of the weight of the current layer is set to a lower bit width than before the quantization bit width; If the smallest If it is not greater than the first set threshold, the quantization target of the weight after optimization of the current layer is set to a lower bit width than before quantization, and the quantization target of the activation after optimization of the current layer is set to a lower bit width than before quantization.

4. The method according to claim 3, characterized in that In the smallest Under the condition that the value is greater than the first set threshold, the process of setting the quantization target of the weight of the current layer to a lower bit width than before quantization includes: In the smallest When the value is greater than the first threshold value: If max(W) is less than or equal to the second set threshold, the quantization target of the weight of the current layer is set to W2, where W represents the weight and 2 represents the bit width after quantization; If max(W) is greater than the second set threshold and less than or equal to the third set threshold, the quantization target of the weight of the current layer is set to W4, where 4 represents the bit width after quantization; If max(W) is greater than the third set threshold and less than or equal to the fourth set threshold, the quantization target of the weight of the current layer is set to W8, where 8 represents the bit width after quantization; The second set threshold is smaller than the third set threshold, and the third set threshold is smaller than the fourth set threshold.

5. The method according to claim 3, characterized in that: In the smallest The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the quantization target of the weight after optimization of the current layer is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, including: In the smallest Under the condition that it is not greater than the first set threshold: If the smallest If the value is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A4, where W represents weight, A represents activation, and the number 4 represents the bit width after quantization, and the fifth set threshold is less than the first set threshold.

6. The method according to claim 5, characterized in that In the smallest The process of setting the quantization target of the weight after optimization of the current layer to a lower bit width than before quantization under the condition that the weight is not greater than the first set threshold, and setting the quantization target of the activation after optimization of the current layer to a lower bit width than before quantization, further includes: In the smallest Under the condition that it is not greater than the first set threshold: If the smallest If it is greater than the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W8, and the quantization target of the activation after optimization of the current layer is set to A8; like is greater than the fifth set threshold and less than or equal to the first set threshold, and If it is less than or equal to the fifth set threshold, the quantization target of the weight after optimization of the current layer is set to W4, and the quantization target of the activation after optimization of the current layer is set to A8.

7. The method according to claim 2, characterized in that The process of quantizing each layer of the target model according to the quantization target of each layer includes: For each layer of the target model: If the quantization target of the current layer indicates to keep activation without quantization and only quantize weights, the weights of the current layer are quantized offline according to the set quantization algorithm; If the quantization target of the current layer indicates that both the optimized weights and the optimized activations of the current layer are quantized, the optimized weights of the current layer are quantized offline according to the set quantization algorithm. At the same time, diag(s) is saved for the online inference process of the target model. Reasoning is performed on the real-time input data and the activation of the current layer is obtained. The activation of the current layer is divided by diag(s) as the optimized activation, and the optimized activation of the current layer is quantized online according to the set quantization algorithm.

8. The method according to claim 7, characterized in that The setting quantization algorithm adopts the following formula: Among them, V i，j Indicates the elements in the weight or activation matrix that need to be quantized, Quant(V i，j ) represents the quantized result, and m represents the quantization bit width; Based on this, if the quantization target of the current layer means that both the optimized weights and the optimized activations of the current layer are quantized, then max(|X j |) for online quantization of the current layer after optimization, and save the max(|X j |) as the max|V in the quantization algorithm j |.

9. A model quantization device, characterized in that: include: A model parsing unit, used to parse the structure of the target model to be quantized, and obtain the weight and activation of each layer in the target model; A quantization target determination unit, configured to determine a quantization target for each layer of the target model according to the weight and activation values ​​of each layer, using each layer as a quantization granularity, wherein the quantization target includes quantization targets for each weight and activation; The quantization processing unit is used to perform quantization processing on each layer of the target model according to the quantization target of each layer.

10. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the model quantization method according to any one of claims 1 to 9.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the model quantization method according to any one of claims 1 to 9 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the model quantization method according to any one of claims 1 to 9 is implemented.

Images