A TorchScript model quantization method, device, equipment and storage medium

Abstract

The application is suitable for the field of model quantification, and provides a TorchScript model quantification method, device and storage medium.The method comprises the following steps: performing quantification processing on original input values of an input TorchScript model to generate target input values, performing quantification processing on first weights of a current network layer of the TorchScript model according to a preset quantification candidate bit width to generate first sensitivity information, performing quantification processing on the first weights based on the first sensitivity information to generate second weights, obtaining first activation values of the current network layer based on the target input values and the second weights, performing quantification processing on the first activation values according to the quantification candidate bit width to generate second sensitivity information, performing quantification processing on the first activation values based on the second sensitivity information to generate second activation values, and outputting a quantified TorchScript model when the current network layer is the last layer of the TorchScript model, so that the quantified TorchScript model maintains high precision and low bit width, thereby ensuring that the quantified TorchScript model can be successfully deployed on an embedded terminal device.

Description

Technical Field

[0001] This invention belongs to the field of model quantization technology, and in particular relates to a TorchScript model quantization method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of deep learning, it has been widely adopted in fields such as speech, image, and natural language processing. Numerous technologies based on neural network models are constantly emerging, leading to a surge in AI chip manufacturers. However, due to the characteristics of neural network models—numerous layers, parameters, and computational demands—deploying these models on chips still faces challenges such as insufficient computing resources, insufficient storage space, and excessive memory consumption. Therefore, it is necessary to optimize and compress neural network models, or adopt fixed-point arithmetic to reduce the high power consumption of floating-point operations, ensuring successful deployment on embedded terminal devices. Traditional 8-bit quantization compression schemes often result in low precision. For example, the precision loss caused by quantization weights is more pronounced for networks with fewer layers, while the precision loss caused by quantization activation is more significant for networks with more layers. Furthermore, traditional 8-bit quantization compression schemes still treat BatchNorm layers in the network as floating-point operations without quantization. Since most networks have a large number of BatchNorm layers, the speedup effect of this incomplete quantization method is not comprehensive.

[0003] Therefore, there is an urgent need for a high-precision quantization compression scheme to quantize and compress neural network models in order to reduce the high power consumption caused by floating-point operations and ensure that they can be successfully deployed on embedded terminal devices. Summary of the Invention

[0004] The purpose of this invention is to provide a TorchScript model quantization method, apparatus, device, and storage medium, aiming to solve the problem that existing technologies cannot provide a high-precision, low-bit-width model quantization method.

[0005] In a first aspect, the present invention provides a TorchScript model quantization method, the method comprising the following steps: The raw input values ​​of the TorchScript model are quantized to generate the target input values; According to the preset quantization candidate bit width, the first weight of the current network layer of the TorchScript model is quantized to obtain the first quantization result, and the first sensitivity information is generated based on the first quantization result. The first weight is quantized based on the first sensitivity information to generate a second weight; Based on the target input value and the second weight, the first activation value of the current network layer is obtained. According to the quantization candidate bit width, the first activation value is quantized to obtain the second quantization result. Based on the second quantization result, the second sensitivity information is generated. The first activation value is quantized based on the second sensitivity information to generate a second activation value; Determine whether the current network layer is the last layer of the TorchScript model. If yes, output the quantized TorchScript model. Otherwise, set the second activation value to the target input value, set the next layer corresponding to the current network layer to the current network layer, update the first sensitivity information according to the second sensitivity information, and jump to the step of quantizing the first weight based on the first sensitivity information to generate the second weight.

[0006] In some embodiments, the step of quantizing the raw input values ​​of the input TorchScript model to generate target input values ​​includes: Histogram statistics are performed on the original input values ​​to obtain the first histogram. Based on the first histogram, several groups of second histograms are obtained. Calculate the KL divergence value of the second histogram, determine the first quantization parameter based on the KL divergence value, and quantize the original input value according to the first quantization parameter to obtain the target input value.

[0007] In some embodiments, the step of quantizing the first weight of the current network layer of the TorchScript model according to a preset quantization candidate bit width to obtain a first quantization result, and generating first sensitivity information based on the first quantization result, includes: Based on several precision values ​​of the quantization candidate bit width, the first weight is quantized respectively to obtain the first quantization result; Based on the first quantization result, calculate the different sensitivities of the first weight to several precision values, and generate the first sensitivity information based on the obtained different sensitivities.

[0008] In some embodiments, the step of quantizing the first weight based on the first sensitivity information to generate a second weight includes: The quantization bit width is obtained from the first sensitivity information, and the first weight is quantized according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate quantization weights. Calculate the first similarity distance between the first weight and each of the quantized weights, and determine the weight quantization parameters based on the first similarity distance; The first weight of the current network layer is quantized according to the weight quantization parameters to obtain the second weight.

[0009] In some embodiments, the first activation value is quantized according to a preset quantization candidate bit width to obtain a second quantization result, and the step of generating second sensitivity information based on the second quantization result includes: Based on several precision values ​​of the quantization candidate bit width, the first activation value is quantized to obtain a second quantization result. Based on the second quantization result, the different sensitivities of the first activation value to several precision values ​​are calculated, and the second sensitivity information is generated according to the obtained different sensitivities.

[0010] In some embodiments, the step of quantizing the first activation value based on the second sensitivity information to generate the second activation value includes: The quantization bit width is obtained from the second sensitivity information, and the first activation value is quantized according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate a quantized activation value. Calculate the second similarity distance between the first activation value and each quantized activation value, and determine the activation quantization parameters based on the second similarity distance; The first activation value of the current network layer is quantized according to the activation quantization parameters to obtain the second activation value; If the output error of the current network layer is greater than a preset precision threshold, the process jumps to the step of obtaining the quantization bit width from the second sensitivity information to reset the quantization bit width and continue searching for and activating the quantization parameter; otherwise, the second activation value is output.

[0011] Secondly, the present invention provides a TorchScript model quantization device, the device comprising: The input quantization unit is used to quantize the raw input values ​​of the TorchScript model to generate the target input values; The weight quantization unit is used to quantize the first weight of the current network layer of the TorchScript model according to the preset quantization candidate bit width, to obtain the first quantization result, and to generate the first sensitivity information based on the first quantization result. The weight generation unit is used to quantize the first weight based on the first sensitivity information to generate a second weight; The activation value quantization unit is used to obtain the first activation value of the current network layer based on the target input value and the second weight, quantize the first activation value according to the quantization candidate bit width to obtain the second quantization result, and generate second sensitivity information based on the second quantization result. An activation value generation unit is used to quantize the first activation value based on the second sensitivity information to generate a second activation value; The network layer determination unit is used to determine whether the current network layer is the last layer of the TorchScript model. If it is, the quantized TorchScript model is output; otherwise, the second activation value is set as the target input value, the next layer corresponding to the current network layer is set as the current network layer, the first sensitivity information is updated according to the second sensitivity information, and the weight generation unit is triggered to quantize the first weight based on the first sensitivity information to generate the second weight.

[0012] In some embodiments, the input quantization unit includes: The histogram acquisition unit is used to perform histogram statistics on the original input values ​​to obtain a first histogram, and to group several groups of second histograms based on the first histogram. A parameter determination unit is used to calculate the KL divergence value of the second histogram and determine a first quantization parameter based on the KL divergence value; An input quantization subunit is used to quantize the original input value according to the first quantization parameter to obtain the target input value.

[0013] Thirdly, the present invention also provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0014] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0015] In this embodiment of the invention, during the quantization of the TorchScript model, the original input value of the TorchScript model is quantized to generate a target input value. Based on a preset quantization candidate bit width, the first weight of the current network layer of the TorchScript model is quantized to obtain a first quantization result. Based on the first quantization result, first sensitivity information is generated. Based on the first sensitivity information, the first weight is quantized to generate a second weight. Based on the target input value and the second weight, the first activation value of the current network layer is obtained. Based on the quantization candidate bit width, the first activation value is quantized to obtain a second quantization result. Based on the second quantization result, second sensitivity information is generated. Based on the second sensitivity information, the first activation value is quantized to generate a second activation value. It is then determined whether the current network layer is the last layer of the TorchScript model. If so, the quantized TorchScript model is output; otherwise, the second activation value is set as the target input value, and the next layer corresponding to the current network layer is set as the current network layer. The first sensitivity information is updated based on the second sensitivity information, and the model is quantized again. In this way, the quantized TorchScript model can maintain high accuracy while keeping the bit width low. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a TorchScript model quantization method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating step S105 in the aforementioned TorchScript model quantization method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a TorchScript model quantization device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Furthermore, the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. The terms "first," "second," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships, which may change accordingly when the absolute position of the described object changes.

[0019] To keep the following description of the embodiments of the present invention clear and concise, detailed descriptions of some known functions and known components are omitted in this specification.

[0020] The specific implementation of the present invention will be described in detail below with reference to specific embodiments: Figure 1 The following is an implementation flow of a TorchScript model quantization method according to an embodiment of the present invention. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: In step S101, the original input values ​​of the TorchScript model are quantized to generate the target input values.

[0021] This invention is applicable to computing devices, such as personal computers and servers, for quantizing TorchScript models. In this invention, the original input value of the TorchScript model is the activation value of the TorchScript model's input layer. Quantizing the original input value, i.e., the activation value, yields the target input value.

[0022] In step S102, the first weight of the current network layer of the TorchScript model is quantized according to the preset quantization candidate bit width to obtain the first quantization result, and the first sensitivity information is generated based on the first quantization result.

[0023] In step S103, the first weight is quantized based on the first sensitivity information to generate the second weight.

[0024] In this embodiment of the invention, the first weight is the unquantized weight of the network layer of the TorchScript model, and the second weight is the quantized weight of the network layer of the TorchScript model.

[0025] In step S104, based on the target input value and the second weight, the first activation value of the current network layer is obtained. According to the quantization candidate bit width, the first activation value is quantized to obtain the second quantization result. The second sensitivity information is generated based on the second quantization result.

[0026] In this embodiment of the invention, the target input value is set as the input of the current network layer. Based on the target input value, network forward propagation calculation is performed to obtain the first activation value of the current network layer. The first activation value is the activation value to be quantized of the network layer of the TorchScript model.

[0027] In step S105, the first activation value is quantized based on the second sensitivity information to generate a second activation value.

[0028] In this embodiment of the invention, the second activation value is the quantized activation value of the network layer of the TorchScript model.

[0029] In step S106, it is determined whether the current network layer is the last layer of the TorchScript model. If yes, step S108 is executed; otherwise, step S107 is executed.

[0030] In this embodiment of the invention, when determining whether the current network layer is the last layer of the TorchScript model, specifically, the number of the current network layer is compared with the total number of network layers in the TorchScript model. If the number of the current network layer is less than the total number of network layers in the TorchScript model, then the current network layer is determined not to be the last layer of the TorchScript model, and step S107 is executed. If the number of the current network layer is equal to the total number of network layers in the TorchScript model, then the current network layer is determined to be the last layer of the TorchScript model, and step S108 is executed.

[0031] In step S107, the second activation value is set as the target input value, the next layer corresponding to the current network layer is set as the current network layer, the first sensitivity information is updated according to the second sensitivity information, and then the process jumps to step S103.

[0032] In this embodiment of the invention, when the current network layer is not the last layer of the TorchScript model, the second activation value output by the current network layer is set as the target input value of the next network layer, the next layer corresponding to the current network layer is taken as the current network layer, and the first sensitivity information is updated according to the second sensitivity information. Specifically, the quantization bit width in the first sensitivity information is reset to the quantization bit width in the second sensitivity information, and the process jumps to step S103 to continue execution to continue quantizing the TorchScript model. Specifically, when the quantization bit width in the first weight is reset to the quantization bit width in the second sensitivity information, for example, if the quantization bit width in the first weight is 4 bits and the quantization bit width in the second sensitivity information is 8 bits, the quantization bit width in the first sensitivity information is set from 4 bits to 8 bits to update the first sensitivity information.

[0033] In step S108, the quantized TorchScript model is output.

[0034] In this embodiment of the invention, when the current network layer is the last layer of the TorchScript model, the quantization process is completed, and the quantized TorchScript model is output, thus realizing the quantization of the TorchScript model.

[0035] In this embodiment of the invention, during the quantization of the TorchScript model, the original input value of the TorchScript model is quantized to generate a target input value. Based on a preset quantization candidate bit width, the first weight of the current network layer of the TorchScript model is quantized to obtain a first quantization result. Based on the first quantization result, first sensitivity information is generated. Based on the first sensitivity information, the first weight is quantized to generate a second weight. Based on the target input value and the second weight, the first activation value of the current network layer is obtained. Based on the quantization candidate bit width, the first activation value is quantized to obtain a second quantization result. Based on the second quantization result, second sensitivity information is generated. Based on the second sensitivity information, the first activation value is quantized to generate a second activation value. It is then determined whether the current network layer is the last layer of the TorchScript model. If so, the quantized TorchScript model is output; otherwise, the second activation value is set as the target input value, and the next layer corresponding to the current network layer is set as the current network layer. The first sensitivity information is updated based on the second sensitivity information, and the model is quantized again. In this way, the quantized TorchScript model can maintain high accuracy while keeping the bit width low.

[0036] In some embodiments, step S101, when quantizing the original input values ​​of the input TorchScript model to generate the target input values, can be specifically implemented through the following steps: (1) Perform histogram statistics on the original input values ​​to obtain the first histogram; In this embodiment of the invention, the original input values ​​are statistically analyzed in the form of a histogram to obtain a first histogram, which visually displays the data distribution of the original input values.

[0037] (2) Based on the first histogram, several groups of second histograms are obtained by grouping. In this embodiment of the invention, the first histogram is divided into several groups of second histograms according to a preset group interval.

[0038] (3) Calculate the KL divergence value of the second histogram and determine the first quantization parameter based on the KL divergence value; In this embodiment of the invention, the KL divergence value of the second histogram is calculated for each quantization parameter within a preset quantization parameter range. Based on the KL divergence value, a first quantization parameter is determined within the preset quantization parameter range. Specifically, the quantization parameter with the smallest calculated KL divergence value is selected as the first quantization parameter.

[0039] (4) Quantize the original input value according to the first quantization parameter to obtain the target input value.

[0040] In this embodiment of the invention, the original input value is quantized based on the first quantization parameter to obtain the target input value, which will continue to be transmitted down to the next network layer.

[0041] In this embodiment of the invention, the quantization of the original input value is achieved through the above steps (1)-(4), thereby ensuring that the quantized original input value maintains a high level of accuracy.

[0042] In some embodiments, when step S102 quantizes the first weight of the current network layer of the TorchScript model according to a preset quantization candidate bit width to obtain a first quantization result, and generates first sensitivity information based on the first quantization result, it can be specifically implemented through the following steps: (1) Based on several precision values ​​of the quantization candidate bit width, the first weight is quantized respectively to obtain the first quantization result.

[0043] In this embodiment of the invention, the quantization candidate bit width is a number of bits, which is the precision value. The precision value of the quantization candidate bit width can be 16 bits, 8 bits, or 4 bits. Based on the precision values ​​of the quantization candidate bit width, the first weight is quantized to obtain a first quantization result. Then, the first quantization result can be generated based on the first weight and the precision values. Specifically, first precision quantization can be performed based on the first weight to generate a first precision quantization result. Then, second precision quantization can be performed based on the first precision quantization result to generate a second precision quantization result. Then, third precision quantization can be performed based on the second precision quantization result to generate a third precision quantization result. Finally, the aforementioned first quantization result is generated based on the first precision quantization result, the second precision quantization result, and the third precision quantization result. For example, 16-bit quantization can be performed based on the first weight to generate a 16-bit quantization result. Then, 8-bit quantization can be performed based on the 16-bit quantization result to generate an 8-bit quantization result. Then, 4-bit quantization can be performed based on the 8-bit quantization result to generate a 4-bit quantization result. The first quantization result is generated based on the 16-bit quantization result, the 8-bit quantization result, and the 4-bit quantization result.

[0044] (2) Calculate the different sensitivities of the first weight to several precision values ​​based on the first quantization result, and generate the first sensitivity information based on the obtained different sensitivities.

[0045] In this embodiment of the invention, when calculating the different sensitivities of the first weight to several precision values ​​based on the first quantization result, specifically, based on the aforementioned steps, a first cosine similarity is calculated between the first precision quantization result and the second precision quantization result. If the first cosine similarity is greater than a preset threshold, it indicates that the first weight is less sensitive to the second precision quantization than to the first precision quantization. A second cosine similarity is then calculated between the second precision quantization result and the third precision quantization result. If the second cosine similarity is greater than a preset threshold, it indicates that the first weight is less sensitive to the third precision quantization than to the second precision quantization. Therefore, it is concluded that the first weight has the lowest sensitivity to the third precision quantization, and the third precision can be set as the quantization bit width. First sensitivity information is then generated based on the first cosine similarity and the second cosine similarity. For example, the first cosine similarity between the 16-bit quantization result and the 8-bit quantization result is calculated. If the first cosine similarity is greater than a preset threshold, it means that the first weight is less sensitive to 8-bit quantization than to 16-bit quantization. The second cosine similarity between the 8-bit quantization result and the 4-bit quantization result is calculated. If the second cosine similarity is greater than a preset threshold, it means that the first weight is less sensitive to 4-bit quantization than to 8-bit quantization. Thus, it is concluded that the first weight is least sensitive to 4-bit quantization, and 4 bits can be set as the quantization bit width. Therefore, the different sensitivity information of the first weight to several precision values, i.e., the first sensitivity information, can be obtained based on the first cosine similarity and the second cosine similarity.

[0046] In some embodiments, step S103, when generating the second weight by quantizing the first weight based on the first sensitivity information, may optionally be implemented through the following steps: (1) Obtain the quantization bit width from the first sensitivity information, and quantize the first weight according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate the quantization weight.

[0047] In this embodiment of the invention, the quantization parameter range is the set of quantization parameters corresponding to the quantization bit width. The weight quantization parameter is determined within the quantization parameter range. Based on each quantization parameter in the quantization parameter range, the first weight is quantized to generate several quantization weights.

[0048] (2) Calculate the first similarity distance between the first weight and each quantized weight, and determine the weight quantization parameters based on the first similarity distance.

[0049] In this embodiment of the invention, a first similarity distance is calculated between the first weight and each quantization weight, and the quantization parameter corresponding to the quantization weight with the smallest first similarity distance to the first weight is set as the weight quantization parameter. Preferably, the first similarity distance can be the mean square error between the first weight and each quantization weight.

[0050] (3) Quantize the first weight of the current network layer according to the weight quantization parameter to obtain the second weight.

[0051] In this embodiment of the invention, the first weight is quantized based on the weight quantization parameter to obtain the quantized second weight, thereby realizing the quantization of the first weight.

[0052] In some embodiments, when step S104 quantizes the first activation value according to a preset quantization candidate bit width to obtain a second quantization result, and generates second sensitivity information based on the second quantization result, it can be specifically implemented through the following steps: (1) Based on several precision values ​​of the quantization candidate bit width, the first activation value is quantized to obtain the second quantization result.

[0053] In this embodiment of the invention, the quantization candidate bit width is a number of bits, i.e., a precision value. The precision values ​​of the quantization candidate bit width can be 16 bits, 8 bits, or 4 bits. When the first activation value is quantized based on the precision values ​​of the quantization candidate bit width to obtain a second quantization result, the second quantization result can be generated based on the first activation value and the precision values. Specifically, a first precision quantization can be performed based on the first activation value to generate a first precision quantization result. Then, a second precision quantization can be performed based on the first precision quantization result to generate a second precision quantization result. A third precision quantization can then be performed based on the second precision quantization result to generate a third precision quantization result. The aforementioned second quantization result is generated based on the first precision quantization result, the second precision quantization result, and the third precision quantization result. For example, a 16-bit quantization can be performed based on the first activation value to generate a 16-bit quantization result. Then, an 8-bit quantization can be performed based on the 16-bit quantization result to generate an 8-bit quantization result. Finally, a 4-bit quantization can be performed based on the 8-bit quantization result to generate a 4-bit quantization result. The second quantization result is composed of the 16-bit quantization result, the 8-bit quantization result, and the 4-bit quantization result.

[0054] (2) Calculate the different sensitivities of the first activation value to several precision values ​​based on the second quantization result, and generate second sensitivity information based on the obtained different sensitivities. In this embodiment of the invention, when calculating the different sensitivities of the first activation value to several precision values ​​based on the second quantization result, specifically, based on the aforementioned steps, the third cosine similarity between the first precision quantization result and the second precision quantization result is calculated. If the third cosine similarity is greater than a preset threshold, it indicates that the sensitivity of the first activation value to the second precision quantization is lower than that to the first precision quantization. The fourth cosine similarity between the second precision quantization result and the third precision quantization result is calculated. If the fourth cosine similarity is greater than a preset threshold, it indicates that the sensitivity of the first activation value to the third precision quantization is lower than that to the second precision quantization. Thus, it is concluded that the first activation value has the lowest sensitivity to the third precision quantization, and the third precision can be set as the quantization bit width. Finally, the second sensitivity information is generated based on the third cosine similarity and the fourth cosine similarity. For example, the third cosine similarity between the 16-bit quantization result and the 8-bit quantization result is calculated. If the third cosine similarity is greater than a preset threshold, it means that the first activation value is less sensitive to 8-bit quantization than to 16-bit quantization. The fourth cosine similarity between the 8-bit quantization result and the 4-bit quantization result is calculated. If the fourth cosine similarity is greater than a preset threshold, it means that the first activation value is less sensitive to 4-bit quantization than to 8-bit quantization. Thus, it is concluded that the first activation value is least sensitive to 4-bit quantization, and 4 bits can be set as the quantization bit width. Therefore, the different sensitivity information of the first activation value to several precision values, i.e., the second sensitivity information, can be obtained based on the third cosine similarity and the fourth cosine similarity.

[0055] Figure 2 The implementation flow of S105 in the foregoing embodiment is shown. For ease of explanation, only the parts related to the embodiments of the present invention are shown, and are described in detail below: Step S201: Obtain the quantization bit width from the second sensitivity information, and quantize the first activation value according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate a quantized activation value. In this embodiment of the invention, the quantization parameter range is the set of quantization parameters corresponding to the quantization bit width. The activation value quantization parameter is determined within the quantization parameter range. Specifically, based on each quantization parameter of the quantization parameter range, the first activation value is quantized to generate several quantized activation values.

[0056] Step S202: Calculate the second similarity distance between the first activation value and each quantized activation value, and determine the activation quantization parameters based on the second similarity distance; In this embodiment of the invention, a second similarity distance is calculated between the first activation value and each quantized activation value, and the quantization parameter corresponding to the quantized activation value with the smallest second similarity distance is set as the activation value quantization parameter. Preferably, the second similarity distance can be the mean square error between the first activation value and each quantized activation value.

[0057] Step S203: Quantize the first activation value of the current network layer according to the activation quantization parameters to obtain the second activation value; Step S204: Determine whether the output error of the current network layer is greater than the preset accuracy threshold. If yes, proceed to step S205; otherwise, jump to step S201 to continue execution.

[0058] In this embodiment of the invention, based on the second weight and the second activation value, the output error of the current network layer is checked for accuracy. If the output error of the current network layer is greater than a preset accuracy threshold, the quantization bit width is reset to the accuracy value corresponding to the previous level of sensitivity according to the sensitivity level of the first activation value to different accuracy values ​​in the second sensitivity information. Based on the reset quantization bit width, the process jumps to step S201 to optimize the quantization of the first activation value. For example, the quantization bit width is reset from 4 bits to 8 bits, and then the process jumps to step S201 to continue searching for activation quantization parameters. Similarly, if the output error of the current network layer is still greater than the preset accuracy threshold after quantizing the first activation value of the current network layer using the activation quantization parameter determined in the 8-bit quantization parameter range, the quantization bit width is reset from 8 bits to 16 bits, and the process jumps to step S201 until the output error of the current network layer is less than the accuracy threshold, and then the process jumps to step S205.

[0059] Step S205: Output the second activation value.

[0060] In this embodiment of the invention, the output error of the current network layer is checked for accuracy based on the second weight and the second activation value. If the output error of the current network layer is less than the accuracy threshold, the second activation value is output, thereby improving the quantization effect of the first activation value.

[0061] Figure 3 The structure of a TorchScript model quantization device according to an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, including: The input quantization unit 31 is used to quantize the original input values ​​of the input TorchScript model to generate the target input values; The weight quantization unit 32 is used to quantize the first weight of the current network layer of the TorchScript model according to the preset quantization candidate bit width, to obtain the first quantization result, and to generate the first sensitivity information based on the first quantization result. The weight generation unit 33 is used to quantize the first weight based on the first sensitivity information to generate the second weight; The activation value quantization unit 34 is used to obtain the first activation value of the current network layer based on the target input value and the second weight, quantize the first activation value according to the quantization candidate bit width to obtain the second quantization result, and generate the second sensitivity information based on the second quantization result. The activation value generation unit 35 is used to quantize the first activation value based on the second sensitivity information to generate a second activation value; The network layer judgment unit 36 ​​is used to determine whether the current network layer is the last layer of the TorchScript model. If it is, the quantized TorchScript model is output. Otherwise, the second activation value is set as the target input value, the next layer corresponding to the current network layer is set as the current network layer, the first sensitivity information is updated according to the second sensitivity information, and the weight generation unit is triggered to quantize the first weight based on the first sensitivity information to generate the second weight, so as to continue to quantize the TorchScript model until the quantization of the TorchScript model is finally completed.

[0062] In some embodiments, the input quantization unit 31 may include: The histogram acquisition unit is used to perform histogram statistics on the original input values ​​to obtain the first histogram, and to group the first histogram to obtain several groups of second histograms. The parameter determination unit is used to calculate the KL divergence value of the second histogram and determine the first quantization parameter based on the KL divergence value; The input quantization subunit is used to quantize the original input value according to the first quantization parameter to obtain the target input value.

[0063] In this embodiment of the invention, for the sake of convenience and brevity, only the division of the above-described functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to achieve all or part of the functions described above. Each unit and module of the device can be implemented by corresponding hardware or software units. Each unit and module can be an independent hardware or software unit, or it can be integrated into a single hardware or software unit, which is not intended to limit the invention. In addition, the specific names of each functional unit and module are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the device can be referred to the corresponding description in the foregoing method embodiments, and will not be repeated here.

[0064] Figure 4 The structure of a computing device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0065] The computing device 4 of this embodiment includes a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, it implements the steps described in the various TorchScript model quantization method embodiments above, for example... Figure 1 The steps S101 to S108 are shown. Alternatively, when the processor 40 executes the computer program 42, it implements the functions of each unit in the above-described device embodiment, for example... Figure 3 The functions of each unit are shown.

[0066] When the processor 40 in the computing device 4 executes the computer program 42 to implement the TorchScript model quantization method, the steps implemented can be referred to the description of the aforementioned method embodiment, and will not be repeated here.

[0067] In one embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps described in the TorchScript model quantization method embodiment. For example... Figure 1 The steps S101 to S108 are shown. Alternatively, when the computer program is executed by a processor, it implements the functions of each unit in the above-described apparatus embodiment, for example... Figure 3 The functions of each unit are shown.

[0068] The computer-readable storage medium of this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0069] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the scope of disclosure involved in the above embodiments is not limited to technical solutions formed by specific combinations of the above technical features, but should also cover other technical solutions formed by arbitrary combinations of the above technical features or their equivalent features without departing from the above-disclosed concept. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0070] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A TorchScript model quantization method, characterized in that, The method includes the following steps: The raw input values ​​of the TorchScript model are quantized to generate the target input values; According to the preset quantization candidate bit width, the first weight of the current network layer of the TorchScript model is quantized to obtain the first quantization result, and the first sensitivity information is generated based on the first quantization result. The first weight is quantized based on the first sensitivity information to generate a second weight; Based on the target input value and the second weight, the first activation value of the current network layer is obtained. According to the quantization candidate bit width, the first activation value is quantized to obtain the second quantization result. Based on the second quantization result, the second sensitivity information is generated. The first activation value is quantized based on the second sensitivity information to generate a second activation value; Determine whether the current network layer is the last layer of the TorchScript model. If yes, output the quantized TorchScript model. Otherwise, set the second activation value to the target input value, set the next layer corresponding to the current network layer to the current network layer, update the first sensitivity information according to the second sensitivity information, and jump to the step of quantizing the first weight based on the first sensitivity information to generate the second weight.

2. The method as described in claim 1, characterized in that, The steps for quantizing the raw input values ​​of the TorchScript model to generate the target input values ​​include: Histogram statistics are performed on the original input values ​​to obtain the first histogram. Based on the first histogram, several groups of second histograms are obtained. Calculate the KL divergence value of the second histogram, determine the first quantization parameter based on the KL divergence value, and quantize the original input value according to the first quantization parameter to obtain the target input value.

3. The method as described in claim 1, characterized in that, The steps of quantizing the first weight of the current network layer of the TorchScript model according to the preset quantization candidate bit width to obtain the first quantization result, and generating the first sensitivity information based on the first quantization result, include: Based on several precision values ​​of the quantization candidate bit width, the first weight is quantized respectively to obtain the first quantization result; Based on the first quantization result, calculate the different sensitivities of the first weight to several precision values, and generate the first sensitivity information based on the obtained different sensitivities.

4. The method as described in claim 1, characterized in that, The step of quantizing the first weight based on the first sensitivity information to generate the second weight includes: The quantization bit width is obtained from the first sensitivity information, and the first weight is quantized according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate quantization weights. Calculate the first similarity distance between the first weight and each of the quantized weights, and determine the weight quantization parameters based on the first similarity distance; The first weight of the current network layer is quantized according to the weight quantization parameters to obtain the second weight.

5. The method as described in claim 1, characterized in that, The steps of quantizing the first activation value according to the preset quantization candidate bit width to obtain the second quantization result, and generating the second sensitivity information based on the second quantization result include: Based on several precision values ​​of the quantization candidate bit width, the first activation value is quantized to obtain a second quantization result. Based on the second quantization result, the different sensitivities of the first activation value to several precision values ​​are calculated, and the second sensitivity information is generated according to the obtained different sensitivities.

6. The method as described in claim 1, characterized in that, The steps of quantizing the first activation value based on the second sensitivity information to generate the second activation value include: The quantization bit width is obtained from the second sensitivity information, and the first activation value is quantized according to each quantization parameter in the quantization parameter range corresponding to the quantization bit width to generate a quantized activation value. Calculate the second similarity distance between the first activation value and each quantized activation value, and determine the activation quantization parameters based on the second similarity distance; The first activation value of the current network layer is quantized according to the activation quantization parameters to obtain the second activation value; If the output error of the current network layer is greater than a preset precision threshold, the process jumps to the step of obtaining the quantization bit width from the second sensitivity information to reset the quantization bit width and continue searching for and activating the quantization parameter; otherwise, the second activation value is output.

7. A TorchScript model quantization device, characterized in that, The device includes: The input quantization unit is used to quantize the raw input values ​​of the TorchScript model to generate the target input values; The weight quantization unit is used to quantize the first weight of the current network layer of the TorchScript model according to the preset quantization candidate bit width, to obtain the first quantization result, and to generate the first sensitivity information based on the first quantization result. The weight generation unit is used to quantize the first weight based on the first sensitivity information to generate a second weight; The activation value quantization unit is used to obtain the first activation value of the current network layer based on the target input value and the second weight, quantize the first activation value according to the quantization candidate bit width to obtain the second quantization result, and generate second sensitivity information based on the second quantization result. An activation value generation unit is used to quantize the first activation value based on the second sensitivity information to generate a second activation value; The network layer determination unit is used to determine whether the current network layer is the last layer of the TorchScript model. If it is, the quantized TorchScript model is output; otherwise, the second activation value is set as the target input value, the next layer corresponding to the current network layer is set as the current network layer, the first sensitivity information is updated according to the second sensitivity information, and the weight generation unit is triggered to quantize the first weight based on the first sensitivity information to generate the second weight.

8. The apparatus as claimed in claim 7, characterized in that, The input quantization unit includes: The histogram acquisition unit is used to perform histogram statistics on the original input values ​​to obtain a first histogram, and to group several groups of second histograms based on the first histogram. A parameter determination unit is used to calculate the KL divergence value of the second histogram and determine a first quantization parameter based on the KL divergence value; An input quantization subunit is used to quantize the original input value according to the first quantization parameter to obtain the target input value.

9. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images