Mixed-precision quantization method and apparatus, and device

Abstract

The present application relates to the technical field of computers. Provided are a mixed-precision quantization method and apparatus, and a device. The method is applied to each quantization layer comprised in a quantization model, and comprises: acquiring a plurality of parameter files, wherein each parameter file at least comprises a preset quantization bit width and a preset number of iterations; on the basis of each parameter file, generating an initial quantization layer to obtain a plurality of initial quantization layers; on the basis of a target node comprised in each initial quantization layer, constructing an initial node link to obtain a plurality of initial node links, and on the basis of a quantization result of each initial node link, determining a quantization loss value; and on the basis of the quantization loss value corresponding to each initial node link, determining a target node link, and determining the preset quantization bit width corresponding to the target node link to be a target bit width. The method in the present application improves the automation degree and accuracy in determining the quantization bit width of a quantization model.

Description

Mixed precision quantization method, device and equipment Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a mixed-precision quantization method, apparatus, and device. Background Art

[0002] When deploying deep learning models on hardware such as chips, model quantization is generally required to compress the model due to the large number of model parameters. However, this process often results in a decrease in the model's detection and recognition accuracy. Therefore, to balance parameter compression and accuracy during model quantization, mixed-precision quantization is often used. This means using different quantization bit widths at different layers of the quantized model.

[0003] In the prior art, the quantization bit widths of different layers in a quantization model are primarily determined by presetting parameter thresholds. However, the resulting quantization bit widths can result in poor processing accuracy for the quantized model. In this case, manual secondary adjustments to the quantization bit widths are often required, resulting in a complex and inaccurate method for determining the quantization bit widths of the quantization model.

[0004] Summary of the Invention

[0005] The present application provides a mixed-precision quantization method, apparatus, and device to solve the technical problems in the prior art of the quantization bit width of the quantization model, the complex determination method, and the low accuracy.

[0006] In a first aspect, the present application discloses a mixed precision quantization method, which is applied to each quantization layer included in a quantization model, including:

[0007] Acquire multiple parameter files; wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents a bit width to which a parameter in a quantization layer can be quantized; the number of iterations includes the number of iterations for determining the preset quantization bit width; the preset quantization bit widths included in different parameter files are at least partially different;

[0008] Based on each of the parameter files, an initial quantization layer is generated to obtain a plurality of initial quantization layers; wherein the initial quantization layer represents a quantization layer of a graph model type;

[0009] Based on the target nodes included in each of the initial quantization layers, an initial node link is constructed to obtain a plurality of initial node links; and based on the quantization results of each of the initial node links, a quantization loss value is determined; wherein the target node represents a node including a convolution operator;

[0010] Based on the quantization loss values ​​corresponding to each of the initial node links, a target node link is determined; and the pre-set quantization bit width corresponding to the target node link is determined as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

[0011] In one example, generating an initial quantization layer based on each parameter file to obtain multiple initial quantization layers includes:

[0012] Based on each of the parameter files, an initial quantization model is generated to obtain a plurality of initial quantization models;

[0013] An initial quantization layer in each of the initial quantization models is determined to obtain the multiple initial quantization layers.

[0014] In one example, the initial quantization layer includes multiple types of nodes; and constructing an initial node link based on each target node included in the initial quantization layer to obtain multiple initial node links includes:

[0015] After determining the target node in each of the initial quantization layers, obtaining the previous level node corresponding to the target node and obtaining the next level node corresponding to the target node based on the graph structure corresponding to the initial quantization model;

[0016] An initial node link is constructed based on the upper-level node, the target node, and the lower-level node to obtain the multiple initial node links.

[0017] In one example, the quantization loss value includes an output reconstruction loss value and a rounding loss value; the quantization result includes a quantized output result and a quantized weight parameter; and determining the quantization loss value based on the quantization result of each initial node link includes:

[0018] Obtaining an unquantized output result and a quantized output result corresponding to the initial node link; and performing a first loss calculation on the unquantized output result and the quantized output result to obtain the output reconstruction loss value;

[0019] Obtaining a quantized weight parameter corresponding to the initial node link; and performing inverse quantization processing on the quantized weight parameter to obtain an inverse quantized weight parameter;

[0020] Performing a second loss calculation on the unquantized weight parameter and the dequantized weight parameter corresponding to the initial node link to obtain the rounded loss value;

[0021] The quantization loss value is determined based on the output reconstruction loss value and the rounding loss value.

[0022] In one example, after determining a target node link based on the quantization loss value corresponding to each of the initial node links; and determining a preset quantization bit width corresponding to the target node link as the target bit width, the method further includes:

[0023] Based on the target bit width, the quantization bit widths corresponding to the initial node links except the target node link are updated.

[0024] In one example, the initial quantization model includes multiple quantization layers; after updating the quantization bit widths corresponding to the initial node links other than the target node link based on the target bit width, the method further includes:

[0025] Based on the connection order of each quantization layer in the initial quantization model, the initial quantization layer is re-determined; and the target bit width corresponding to the re-determined initial quantization layer is determined until the target bit width corresponding to all quantization layers included in the initial quantization model is determined.

[0026] In one example, the parameter file further includes the number of tuning iterations; and the method further includes:

[0027] After constructing the target quantization model based on the target bit width, the target quantization model is iteratively optimized based on the number of tuning iterations to obtain an optimized target quantization model; wherein the optimized target quantization model is used for quantization processing.

[0028] In a second aspect, the present application discloses a mixed precision quantization device, comprising:

[0029] an acquisition unit, configured to acquire a plurality of parameter files; wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents a bit width to which a parameter in a quantization layer can be quantized; the number of iterations includes the number of iterations used to determine the preset quantization bit width; and the preset quantization bit widths included in different parameter files are at least partially different;

[0030] A generating unit, configured to generate an initial quantization layer based on each of the parameter files, to obtain a plurality of initial quantization layers; wherein the initial quantization layer represents a quantization layer of a graph model type;

[0031] A construction unit is configured to construct an initial node link based on a target node included in each of the initial quantization layers to obtain a plurality of initial node links; and determine a quantization loss value based on a quantization result of each of the initial node links; wherein the target node represents a node including a convolution operator;

[0032] A determination unit is used to determine a target node link based on the quantization loss value corresponding to each of the initial node links; and determine the preset quantization bit width corresponding to the target node link as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine a target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

[0033] In one example, a generation unit is used to:

[0034] Based on each of the parameter files, an initial quantization model is generated to obtain a plurality of initial quantization models;

[0035] An initial quantization layer in each of the initial quantization models is determined to obtain the multiple initial quantization layers.

[0036] In one example, when the initial quantization layer includes multiple types of nodes, the construction unit includes:

[0037] A construction submodule is configured to, after determining a target node in each of the initial quantization layers, obtain a previous-level node corresponding to the target node and obtain a next-level node corresponding to the target node based on a graph structure corresponding to the initial quantization model;

[0038] An initial node link is constructed based on the upper-level node, the target node, and the lower-level node to obtain the multiple initial node links.

[0039] In one example, when the quantization loss value includes an output reconstruction loss value and a rounding loss value; and the quantization result includes a quantized output result and a quantized weight parameter, the construction unit includes:

[0040] A determination submodule is configured to obtain an unquantized output result and a quantized output result corresponding to the initial node link; and perform a first loss calculation on the unquantized output result and the quantized output result to obtain the output reconstruction loss value;

[0041] Obtaining a quantized weight parameter corresponding to the initial node link; and performing inverse quantization processing on the quantized weight parameter to obtain an inverse quantized weight parameter;

[0042] Performing a second loss calculation on the unquantized weight parameter and the dequantized weight parameter corresponding to the initial node link to obtain the rounded loss value;

[0043] The quantization loss value is determined based on the output reconstruction loss value and the rounding loss value.

[0044] In one example, after determining a target node link based on the quantization loss value corresponding to each of the initial node links; and determining a preset quantization bit width corresponding to the target node link as the target bit width, the apparatus further includes:

[0045] An updating module is configured to update the quantization bit width corresponding to the initial node links except the target node link based on the target bit width.

[0046] In one example, when the initial quantization model includes multiple quantization layers, the updating module is further configured to:

[0047] After updating the quantization bit width corresponding to the initial node links other than the target node link based on the target bit width, the initial quantization layer is re-determined based on the connection order of each quantization layer in the initial quantization model; and the target bit width corresponding to the re-determined initial quantization layer is determined, until the target bit width corresponding to all quantization layers included in the initial quantization model is determined.

[0048] In one example, when the parameter file also includes the number of tuning iterations, the device is further configured to:

[0049] After constructing the target quantization model based on the target bit width, the target quantization model is iteratively optimized based on the number of tuning iterations to obtain an optimized target quantization model; wherein the optimized target quantization model is used for quantization processing.

[0050] In a third aspect, the present application discloses an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0051] The memory stores computer-executable instructions;

[0052] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect.

[0053] In a fourth aspect, the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0054] In a fifth aspect, the present application discloses a computer program product, including a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the computer program is executed by at least one processor, so that the electronic device executes the method described in the first aspect.

[0055] The mixed-precision quantization method, apparatus, and device provided herein can pre-generate multiple parameter files based on the bit widths to which the parameters of the quantization layer can be quantized and the number of iterations determined by a priori methods. In this case, by pre-setting the parameter files, constraints can be provided for determining the optimal bit width, saving time and improving efficiency. After obtaining multiple parameter files, an initial quantization model is generated based on the parameter files, resulting in multiple initial quantization models. The initial quantization layers in the initial quantization models can then be determined, obtaining the initial quantization layers corresponding to each parameter file. Subsequently, an initial node link can be constructed based on the target nodes included in each initial quantization layer, obtaining multiple initial node links. A quantization loss value can be determined based on the quantization results of each initial node link, and a target node link can be determined based on the minimum value of the quantization loss values ​​corresponding to each initial node link. The pre-set quantization bit width corresponding to the target node link can be determined as the target bit width. This target bit width is the optimal bit width to which the parameters of the quantization layer need to be quantized. A target quantization layer can then be determined based on the target bit width. After all target quantization layers corresponding to the quantization model have been determined, the target quantization model can be obtained. This implementation method can simply and automatically determine the optimal bit width for each quantization layer by calculating the quantization loss value, thereby obtaining the optimal target quantization model. This not only improves the efficiency of determining the optimal bit width, but also increases the accuracy and reliability of the determined optimal bit width. Furthermore, by presetting the number of iterations, the influence of the number of iterations on the determination of the optimal bit width is avoided, further improving the accuracy of the determined optimal bit width. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] FIG1 is a schematic diagram of a flow chart of a mixed-precision quantization method provided in an embodiment of the present application;

[0058] FIG2 is a schematic diagram of a flow chart of another mixed-precision quantization method provided in an embodiment of the present application;

[0059] FIG3 is a schematic diagram of updating the quantization layer bit width provided in an embodiment of the present application;

[0060] FIG4 is a schematic diagram of an implementation flow of a mixed-precision quantization method applied to a quantization model provided in an embodiment of the present application;

[0061] FIG5 is a schematic diagram of the structure of a mixed-precision quantization device provided in an embodiment of the present application;

[0062] FIG6 is a schematic diagram of the structure of another mixed-precision quantization device provided in an embodiment of the present application;

[0063] FIG7 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0064] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0066] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0067] At present, in order to compress deep learning models and deploy them on hardware such as chips, a quantization model after mixed precision quantization is usually used to quantize the deep learning models to reduce the number of parameters of the deep learning models, thereby reducing the difficulty of deploying the deep learning models.

[0068] In the prior art, when determining the quantization model of mixed precision quantization, the quantization bit width of each parameter of the deep learning model is mainly determined by pre-setting the parameter threshold. For example, the quantization bit width that does not exceed the pre-set parameter threshold can be determined as the quantization bit width of the parameters of the deep learning model, so that the deep learning model can be quantized according to the determined quantization bit width. However, this implementation method is prone to the situation where the accuracy of the quantized deep learning model does not meet the requirements. In this case, it is necessary to manually adjust the determined quantization bit width a second time.

[0069] Therefore, in the prior art, when determining the quantization bit width, the accuracy is insufficient, the degree of automation is low, and the determination method is complex, resulting in low determination efficiency.

[0070] The mixed-precision quantization method provided in this application aims to design a method for fully automatically determining the optimal bit width corresponding to each quantization layer in the quantization model, to determine the quantization model that can be used for mixed-precision quantization, and to solve the above technical problems of the prior art.

[0071] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] FIG1 is a flow chart of a mixed-precision quantization method provided in an embodiment of the present application. The method can be applied to each quantization layer of a quantization model. As shown in FIG1 , the mixed-precision quantization method includes:

[0073] S101. Obtain multiple parameter files.

[0074] The parameter file includes at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents the bit width to which the parameters in the quantization layer can be quantized; the number of iterations includes the number of iterations when determining the preset quantization bit width; the preset quantization bit widths included in different parameter files are at least partially different.

[0075] In one example, the preset quantization bit width included in the parameter file includes: the quantization bit width for the activation data of the quantization layer, and the quantization bit width for the weight of the quantization layer; the preset number of iterations includes the number of iterations when judging the preset quantization bit width, that is, the number of iterations when judging whether the preset quantization bit width is the optimal quantization bit width.

[0076] In one example, assuming that the bit width of the quantization layer weights can be int4, int8, fp8, and fp16, a corresponding parameter file can be pre-generated for each bit width, resulting in four parameter files. In this case, the number of multiple parameter files obtained can be four. In this case, each parameter file can contain one bit width. For example, parameter file 1 includes the quantization layer weights with a bit width of int4, parameter file 2 includes the quantization layer weights with a bit width of int8, parameter file 3 includes the quantization layer weights with a bit width of fp8, and parameter file 4 includes the quantization layer weights with a bit width of fp16.

[0077] In one example, when the bit width of the weight of the quantization layer can be 4 types and the bit width of the activation data of the quantization layer can be 2 types, the bit width of the activation data and the bit width of the weight can be combined to generate a parameter file, and finally 8 parameter files are obtained.

[0078] In one example, the parameter file can be understood as a parameter file for a quantization model. In this case, the parameter file includes pre-set quantization bit widths for parameters of each quantization layer in the quantization model, as well as a pre-set number of iterations, wherein the number of iterations of each quantization layer is the same.

[0079] In one example, the number of iterations included in the parameter file can be determined a priori, for example, by determining the number of iterations for determining a preset quantization bit width. In this case, different parameter files can contain the same number of iterations. It should be noted that this application does not impose a specific limitation on the number of iterations preset in the parameter file, which is subject to implementation.

[0080] S102 : Generate an initial quantization layer based on each parameter file to obtain multiple initial quantization layers.

[0081] Among them, the initial quantization layer represents the quantization layer of the graph model type.

[0082] In one example, an activation pseudo-quantizer, a weight pseudo-quantizer, and a sampling observer can be inserted into each layer of the deep learning model network structure. The PyTorch fx module is called according to the parameter file to generate an initial quantization model. The initial quantization layer is determined based on the network structure of each layer of the initial quantization model. The initial quantization model is also a graphical model type quantization model.

[0083] In one example, this application does not limit the type of deep learning model, subject to the quantization supported by the neural network model. Also, the types of activation pseudo-quantizers, weight pseudo-quantizers, and sampling observers are not limited, subject to actual needs.

[0084] In one example, after obtaining an initial quantization model of a graph model type, the initial quantization model of the graph model type may include a connection relationship graph consisting of various nodes and a model structure corresponding to each node.

[0085] Among them, each node may include the node attribute information of the node (wherein, the node attribute information may represent the type of the node, for example, the node attribute information may be call_module type, call_function type, placeholder type), the connection relationship of the node (for example, it may include the upper-level node of the node, and the lower-level node of the node), the IP corresponding to the node, etc.

[0086] In one example, the model structure corresponding to a node may include a pseudo-quantizer corresponding to the node (including an activation pseudo-quantizer and a weight pseudo-quantizer), a sampling observer, and other parameters of the node (e.g., tuned hyperparameters), etc.

[0087] S103 : constructing an initial node link based on the target node included in each initial quantization layer to obtain multiple initial node links; and determining a quantization loss value based on the quantization result of each initial node link.

[0088] The target node represents the node containing the convolution operator.

[0089] In one example, when the type of a node in the initial quantization layer is a call_module type and the node is a convolution operator node, the node is determined as a target node of the initial quantization layer.

[0090] In one example, the quantization result of the initial node link may include the output result after quantization of the initial node link and the weight parameter after quantization. At this time, the quantization loss value can be determined based on the quantization result of the initial node link and the unquantized result (here, the unquantized result includes the output result and weight parameter before quantization of the initial node link).

[0091] In one example, when determining the quantization loss value, iterative calculation can be performed according to a preset number of iterations, thereby avoiding affecting the calculation result of the quantization loss value due to too many or too few iterations.

[0092] In one example, the initial node link corresponds to the parameter file, and in this case, the number of the initial node link corresponds to the number of the parameter file.

[0093] S104 : Determine a target node link based on the quantization loss value corresponding to each initial node link; and determine the preset quantization bit width corresponding to the target node link as the target bit width.

[0094] Among them, the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct the target quantization model; the target quantization model represents the model for realizing mixed precision quantization.

[0095] At this point, the initial node link with the smallest quantization loss value among the initial node links can be determined as the target node link, and the pre-set quantization bit width corresponding to the target node link can be determined as the optimal bit width, thereby determining the optimal bit width from the bit widths to which multiple parameters can be quantized. At this point, the target quantization model can be determined based on the optimal bit width, and the deep learning model can be quantized based on the target quantization model to balance the compression and accuracy of the parameters of the deep learning model during the quantization process.

[0096] As can be seen from the above description, the present application can generate multiple parameter files in advance based on the bit widths to which the parameters of the quantization layer can be quantized and the number of iterations determined according to a priori methods. In this case, by presetting the parameter files, constraints can be provided for determining the optimal bit width, saving time and improving efficiency. After obtaining multiple parameter files, an initial quantization model is generated based on the parameter files, resulting in multiple initial quantization models. At this point, the initial quantization layers in the initial quantization models can be determined, obtaining the initial quantization layers corresponding to each parameter file. Next, an initial node link can be constructed based on the target nodes included in each initial quantization layer, obtaining multiple initial node links. At this point, a quantization loss value can be determined based on the quantization results of each initial node link, and a target node link can be determined based on the minimum value of the quantization loss values ​​corresponding to each initial node link. At this point, the pre-set quantization bit width corresponding to the target node link can be determined as the target bit width. In this case, this target bit width is the optimal bit width to which the parameters of the quantization layer need to be quantized. Subsequently, a target quantization layer can be determined based on this target bit width. After all target quantization layers corresponding to the quantization model have been determined, the target quantization model can be obtained. This implementation method can simply and automatically determine the optimal bit width for each quantization layer by calculating the quantization loss value, thereby obtaining the optimal target quantization model. This not only improves the efficiency of determining the optimal bit width, but also increases the accuracy and reliability of the determined optimal bit width. Furthermore, by presetting the number of iterations, the influence of the number of iterations on the determination of the optimal bit width is avoided, further improving the accuracy of the determined optimal bit width.

[0097] FIG2 is a flow chart of another mixed-precision quantization method provided in an embodiment of the present application. As shown in FIG2 , the mixed-precision quantization method includes:

[0098] S201. Obtain multiple parameter files.

[0099] The parameter file includes at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents the bit width to which the parameters in the quantization layer can be quantized; the number of iterations includes the number of iterations when determining the preset quantization bit width; the preset quantization bit widths included in different parameter files are at least partially different.

[0100] In an example, this step can refer to the content described in S101 above, and will not be described in detail here.

[0101] S202 : Generate an initial quantization model based on each parameter file to obtain multiple initial quantization models.

[0102] The initial quantization model represents a quantization model of the graph model type.

[0103] In one example, you can call the PyTorch fx module based on a parameter file to generate an initial quantized model of the graphical model type. You can then save multiple initial quantized models generated based on multiple parameter files into a quantized model list.

[0104] S203 : Determine an initial quantization layer in each initial quantization model to obtain multiple initial quantization layers.

[0105] In one example, each layer of the network structure in the initial quantization model can be determined as the initial quantization layer one by one. In the case of multiple initial quantization models, multiple initial quantization layers can be obtained. At this time, the multiple initial quantization layers are corresponding quantization layers.

[0106] In the above implementation, an initial quantization model can be generated based on a pre-generated parameter file that includes the bit widths to which the parameters of each quantization layer can be quantized. Then, the initial quantization layers are determined one by one from the initial quantization model, and the optimal bit widths to which each quantization layer can be quantized are determined. This allows for mixed-precision quantization, allowing for more comprehensive and accurate determination of the quantization bit widths of each quantization layer in the quantization model, thereby ensuring the quantization accuracy of the quantization model. Furthermore, the parameter file can be used to increase the speed and efficiency of determining the quantization bit width of the quantization model.

[0107] In one example, the initial quantization layer includes multiple types of nodes (see the content described in S102 above for details). At this time, an initial node link can be constructed based on the target nodes included in each initial quantization layer to obtain multiple initial node links, specifically including the processes described in S204 to S205.

[0108] S204 : After determining the target node in each initial quantization layer, based on the graph structure corresponding to the initial quantization model, obtain the previous level node corresponding to the target node and obtain the next level node corresponding to the target node.

[0109] The target node represents the node containing the convolution operator.

[0110] S205: Construct an initial node link based on the previous level node, the target node, and the next level node to obtain multiple initial node links.

[0111] S206: Determine a quantization loss value based on the quantization result of each initial node link.

[0112] From the above description, it can be seen that the present application can construct an initial node link with the node containing the convolution operator as the center, and determine the optimal bit width through the quantization loss value corresponding to the initial node link, so that the quantization accuracy of the quantization layer can be evaluated through the convolution operator with a large computational amount, which can reduce the number of nodes involved in the quantization loss calculation, save computing resources, and improve performance.

[0113] In one example, the quantization loss value includes an output reconstruction loss value and a rounding loss value; the quantization result includes a quantized output result and a quantized weight parameter. At this time, when determining the quantization loss value based on the quantization result of each initial node link, the following process can be referred to.

[0114] First, the unquantized output result and the quantized output result corresponding to the initial node link are obtained, and a first loss calculation is performed on the unquantized output result and the quantized output result to obtain an output reconstruction loss value.

[0115] In one example, a forward hook can be registered for the initial node link (for example, calling the register_forward_hook function in pytorch) to obtain the output results of the initial node link, including the unquantized output results and the quantized output results.

[0116] In one example, after obtaining the unquantized output result and the quantized output result corresponding to the initial node link, the first loss calculation can be performed according to the following formula (1), and the output reconstruction loss value is obtained after calculation.

[0117] Among them, pred represents the quantized output result corresponding to the initial node link, tgt represents the unquantized output result corresponding to the initial node link; n is the product of the dimension values ​​of the vector corresponding to the output result.

[0118] Then, the quantized weight parameters corresponding to the initial node link can be obtained; and the quantized weight parameters can be dequantized to obtain dequantized weight parameters. At this time, a second loss calculation can be performed on the unquantized weight parameters and dequantized weight parameters corresponding to the initial node link to obtain a rounded loss value.

[0119] Specifically, after obtaining the quantized weight parameters corresponding to the initial node links, the quantized weight parameters may be dequantized according to the following formula (2) to obtain dequantized weight parameters.

[0120] Where s is the scaling value (i.e., scale value), W is the unquantized weight parameter, n is the minimum quantization value, p is the maximum quantization value, and h(V) is any differentiable function with a value between 0 and 1, i.e., h(Vi,j )∈{0,1}, and its calculation formula can be shown as the following formula (3).

[0121] in, and γ are hyperparameters.

[0122] After obtaining the inverse quantized weight parameter, a second loss calculation may be performed on the unquantized weight parameter and the inverse quantized weight parameter according to the following formula (4) to obtain a rounding loss value.

[0123] Among them, x represents the activation data, that is, the input data, and Wx represents the product of the weight parameter and the input data. is the F-norm regularization, λ is a hyperparameter, f reg (V) is the normalization term, and the calculation formula of this normalization term is shown in the following formula (5). reg (V)=∑ i,j 1-|2h(V i,j )-1| β (5)

[0124] Among them, β is a hyperparameter, and its value ranges from 20 to 2, and its value decreases with the increase of the number of tuning iterations.

[0125] After that, the quantization loss value can be determined based on the output reconstruction loss value and the rounding loss value.

[0126] In an example, the quantization loss value can be obtained by summing the output reconstruction loss value and the rounding loss value. The output reconstruction loss value and the rounding loss value can also be weighted and summed to obtain the quantization loss value. The output reconstruction loss value and the rounding loss value can also be weighted and summed and then averaged to obtain the quantization loss value, etc. The specific method of determining the quantization loss value is not limited here, and it is subject to implementation.

[0127] In the above implementation, the quantization loss value can be determined by taking into account the rounding loss of the weight and the output reconstruction loss of the deep learning model after quantization, so as to determine the optimal bit width based on the quantization loss value. This can avoid the phenomenon that the output result of the deep learning model does not meet the accuracy requirements due to the determined optimal bit width, and the staff is required to manually perform secondary adjustments, thereby achieving fully automatic and accurate determination of the optimal bit width.

[0128] S207 : Determine a target node link based on the quantization loss value corresponding to each initial node link; and determine the preset quantization bit width corresponding to the target node link as the target bit width.

[0129] Among them, the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct the target quantization model; the target quantization model represents the model for realizing mixed precision quantization.

[0130] In one example, this step can refer to the content described in S104 above, and will not be described in detail here.

[0131] In one example, a target node link is determined based on the quantization loss value corresponding to each initial node link; and after the pre-set quantization bit width corresponding to the target node link is determined as the target bit width, the quantization bit width corresponding to the initial node links other than the target node link can be updated based on the target bit width.

[0132] In one example, FIG3 is a schematic diagram of an updated quantization layer bit width provided in an embodiment of the present application. Assuming that the number of parameter files is 3, the number of initial quantization layers obtained is also 3. At this time, an initial node link can be constructed based on the initial quantization layer, wherein the format of the initial node link can be described as "i node-c node-o node" as shown in FIG3. Among them, "i node" represents the input pseudo-quantization node, that is, the previous level node; "c node" represents the convolution node, that is, the target node; "o node" represents the pseudo-quantization node after convolution, that is, the next level node. In the case that the initial node link is not quantized, the initial node link outputs an unquantized result.

[0133] At this time, based on the three initial quantization layers, the three initial node links constructed can be shown in Figure 3. Among them, initial node link 1 includes "i1 node - c1 node - o1 node"; initial node link 2 includes "i2 node - c2 node - o2 node"; initial node link 3 includes "i3 node - c3 node - o3 node".

[0134] At this time, after quantizing the deep learning model, the initial quantization models corresponding to each initial node link can be obtained, that is, the initial node link 1 shown in Figure 3 corresponds to the initial quantization model 1, the initial node link 2 corresponds to the initial quantization model 2, and the initial node link 3 corresponds to the initial quantization model 3. After obtaining the quantization results and unquantized results corresponding to each initial node link according to the corresponding model, if it is determined that the target node link is the above-mentioned initial node link 2, the quantization bit width corresponding to the initial node links other than the target node link can be updated, that is, the above-mentioned initial node link 1 and initial node link 3 are updated, so that the quantization bit width corresponding to the initial node links other than the target node link are all the target bit width. At this time, the initial node link after updating the quantization bit width can be the initial node link 1 shown in Figure 3, including "i2 node - c2 node - o2 node"; the initial node link 2 includes "i2 node - c2 node - o2 node"; the initial node link 3 includes "i2 node - c2 node - o2 node".

[0135] In one example, the steps S201 to S207 described above are the process of determining the optimal bit width for a quantization layer in an initial quantization model. A general initial quantization model may include multiple quantization layers. In this case, assuming that the initial quantization layer is the first quantization layer of the initial quantization model, then, based on the quantization loss values ​​corresponding to each initial node link, a target node link is determined; and after the preset quantization bit width corresponding to the target node link is determined as the target bit width, the optimal bit widths of other quantization layers in the initial quantization model can be further determined. The specific process is as follows.

[0136] Based on the connection order of each quantization layer in the initial quantization model, the initial quantization layer is re-determined; and the target bit width corresponding to the re-determined initial quantization layer is determined, until the target bit width corresponding to all quantization layers included in the initial quantization model is determined.

[0137] In one example, FIG4 is a schematic diagram of an implementation flow of a mixed-precision quantization method applied to a quantization model provided in an embodiment of the present application. As shown in FIG4 , the method specifically includes the following steps:

[0138] Step 1: Pre-set the parameter file corresponding to each bit width. At this time, the parameter file corresponding to each bit width can be pre-set according to the bit width to which the parameters of each quantization layer of the quantization model can be quantized.

[0139] Step 2: Generate an initial quantization model based on the parameter file and put it into the quantization model list. At this time, the quantization model list may include multiple initial quantization models.

[0140] Step 3: Construct an initial node link. Specifically, the initial node link can be constructed by using the target nodes included in the initial quantization layer in the initial quantization model.

[0141] Step 4: Input the input data into the deep learning model and obtain the unquantized results corresponding to the initial node links.

[0142] Step 5: Input the input data to each initial quantization model and obtain the quantization results corresponding to the initial node links.

[0143] Step 6: Calculate the quantization loss value.

[0144] Step 7: Select the initial node link with the smallest quantization loss value as the target node link, and determine the target bit width.

[0145] Step 8: Replace other initial node links with the target bit width.

[0146] Step 9: Determine a new initial node link, and repeat steps 4 to 8 until the last initial node link.

[0147] In one example, after determining the target bit widths of each quantization layer included in the initial quantization model, the target quantization model can be obtained. At this time, the target quantization model can be iteratively optimized to determine other hyperparameters of the target quantization model. At this time, each parameter file obtained can also include the number of tuning iterations. The number of tuning iterations is the number of tuning iterations determined according to the a priori method. For example, the number of tuning iterations can be 11,000. There is no limit on the number of tuning iterations here, and it is subject to what can be achieved.

[0148] At this time, after constructing the target quantization model based on the target bit width, the target quantization model can be iteratively optimized based on the number of tuning iterations to obtain an optimized target quantization model, wherein the optimized target quantization model is used for quantization processing.

[0149] In the above implementation, the number of tuning iterations can be pre-determined and set by an a priori method, and the target quantization model can be iteratively optimized according to the pre-set number of tuning iterations, so that the obtained optimized target quantization model can be more accurate.

[0150] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0151] FIG5 is a schematic diagram of the structure of a mixed-precision quantization device provided in an embodiment of the present application. As shown in FIG5 , the mixed-precision quantization device 500 includes:

[0152] An acquisition unit 501 is configured to acquire multiple parameter files, wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents the bit width to which parameters within a quantization layer can be quantized; the number of iterations includes the number of iterations used to determine the preset quantization bit width; and the preset quantization bit widths included in different parameter files are at least partially different.

[0153] The generating unit 502 is configured to generate an initial quantization layer based on each parameter file to obtain a plurality of initial quantization layers, wherein the initial quantization layer represents a quantization layer of a graph model type.

[0154] The construction unit 503 is used to construct an initial node link based on the target node included in each initial quantization layer to obtain multiple initial node links; and determine the quantization loss value based on the quantization result of each initial node link; wherein the target node represents the node containing the convolution operator.

[0155] The determination unit 504 is used to determine the target node link based on the quantization loss value corresponding to each initial node link; and determine the preset quantization bit width corresponding to the target node link as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

[0156] FIG6 is a schematic diagram of the structure of another mixed-precision quantization device provided in an embodiment of the present application. As shown in FIG6 , the mixed-precision quantization device 600 includes:

[0157] An acquisition unit 601 is configured to acquire multiple parameter files, wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents the bit width to which parameters within a quantization layer can be quantized; the number of iterations includes the number of iterations used to determine the preset quantization bit width; and the preset quantization bit widths included in different parameter files are at least partially different.

[0158] The generating unit 602 is configured to generate an initial quantization layer based on each parameter file to obtain a plurality of initial quantization layers, wherein the initial quantization layer represents a quantization layer of a graph model type.

[0159] The construction unit 603 is used to construct an initial node link based on the target node included in each initial quantization layer to obtain multiple initial node links; and determine the quantization loss value based on the quantization result of each initial node link; wherein the target node represents a node containing a convolution operator.

[0160] The determination unit 604 is used to determine the target node link based on the quantization loss value corresponding to each initial node link; and determine the preset quantization bit width corresponding to the target node link as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

[0161] In one example, the generating unit 602 is configured to:

[0162] Based on each parameter file, an initial quantization model is generated to obtain multiple initial quantization models;

[0163] An initial quantization layer in each initial quantization model is determined to obtain multiple initial quantization layers.

[0164] In one example, when the initial quantization layer includes multiple types of nodes, the construction unit 603 includes:

[0165] Construction submodule 6031 is used to, after determining the target node in each initial quantization layer, obtain the previous level node corresponding to the target node and obtain the next level node corresponding to the target node based on the graph structure corresponding to the initial quantization model;

[0166] Based on the previous level node, the target node and the next level node, an initial node link is constructed to obtain multiple initial node links.

[0167] In one example, when the quantization loss value includes an output reconstruction loss value and a rounding loss value; and the quantization result includes a quantized output result and a quantized weight parameter, the construction unit 603 includes:

[0168] The determination submodule 6032 is configured to obtain an unquantized output result and a quantized output result corresponding to the initial node link; and perform a first loss calculation on the unquantized output result and the quantized output result to obtain an output reconstruction loss value;

[0169] Obtaining the quantized weight parameters corresponding to the initial node link; and performing inverse quantization processing on the quantized weight parameters to obtain inverse quantized weight parameters;

[0170] Perform a second loss calculation on the unquantized weight parameters and the inverse quantized weight parameters corresponding to the initial node link to obtain a rounded loss value;

[0171] A quantization loss value is determined based on the output reconstruction loss value and the rounding loss value.

[0172] In one example, after determining a target node link based on the quantization loss value corresponding to each initial node link; and determining a preset quantization bit width corresponding to the target node link as the target bit width, the apparatus further includes:

[0173] The updating module 605 is configured to update the quantization bit widths corresponding to the initial node links except the target node link based on the target bit width.

[0174] In one example, when the initial quantization model includes multiple quantization layers, the updating module 605 is further configured to:

[0175] After updating the quantization bit width corresponding to the initial node links except the target node link based on the target bit width, the initial quantization layer is re-determined based on the connection order of each quantization layer in the initial quantization model; and the target bit width corresponding to the re-determined initial quantization layer is determined, until the target bit width corresponding to all quantization layers included in the initial quantization model is determined.

[0176] In one example, when the parameter file also includes the number of tuning iterations, the device is further configured to:

[0177] After constructing a target quantization model based on the target bit width, the target quantization model is iteratively optimized based on the number of tuning iterations to obtain an optimized target quantization model; wherein the optimized target quantization model is used for quantization processing.

[0178] The mixed-precision quantization device provided in this embodiment can be used to execute the above-mentioned mixed-precision quantization method. Its implementation method and technical effects are similar, and this embodiment will not be repeated here.

[0179] FIG7 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As shown in FIG7 , the electronic device 700 includes: a memory 701 and a processor 702 .

[0180] Memory 701 is a memory used to store instructions executable by processor 702.

[0181] The processor 702 is configured to execute the method provided in the above embodiment.

[0182] The electronic device further includes a receiver 703 and a transmitter 704. The receiver 703 is used to receive instructions and data sent by an external device, and the transmitter 704 is used to send instructions and data to the external device.

[0183] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided in the various embodiments described above.

[0184] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of computer-executable instructions from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be an integral part of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist in a communication device as discrete components.

[0185] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0186] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. At least one processor of a device can read the computer program from the computer-readable storage medium, and at least one processor executes the computer program so that the device implements the methods provided in the various embodiments described above.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0188] The modules may be physically separate, for example, installed in different locations on a single device, or installed on different devices, or distributed across multiple network units, or distributed across multiple processors. The modules may also be integrated, for example, installed in the same device, or integrated into a set of codes. The modules may exist in the form of hardware, or in the form of software, or may be implemented in the form of software plus hardware. The present application may select some or all of the modules according to actual needs to achieve the purpose of the present embodiment.

[0189] When each module is implemented as an integrated module in the form of a software function module, it can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing an electronic device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0190] It should be understood that, although the various steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times, and their execution order is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of sub-steps or stages of other steps.

[0191] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0192] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A mixed precision quantization method, characterized in that: Applied to each quantization layer included in the quantization model, the method includes: Acquire multiple parameter files; wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents a bit width to which a parameter in a quantization layer can be quantized; the number of iterations includes the number of iterations for determining the preset quantization bit width; the preset quantization bit widths included in different parameter files are at least partially different; Based on each of the parameter files, an initial quantization layer is generated to obtain a plurality of initial quantization layers; wherein the initial quantization layer represents a quantization layer of a graph model type; Based on the target nodes included in each of the initial quantization layers, an initial node link is constructed to obtain a plurality of initial node links; and based on the quantization results of each of the initial node links, a quantization loss value is determined; wherein the target node represents a node including a convolution operator; Based on the quantization loss values ​​corresponding to each of the initial node links, a target node link is determined; and the pre-set quantization bit width corresponding to the target node link is determined as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine the target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

2. The method according to claim 1, characterized in that The generating of an initial quantization layer based on each of the parameter files to obtain a plurality of initial quantization layers includes: Based on each of the parameter files, an initial quantization model is generated to obtain a plurality of initial quantization models; An initial quantization layer in each of the initial quantization models is determined to obtain the multiple initial quantization layers.

3. The method according to claim 2, characterized in that The initial quantization layer includes multiple types of nodes; The initial node link is constructed based on the target node included in each of the initial quantization layers to obtain multiple initial node links, including: After determining the target node in each of the initial quantization layers, obtaining the previous level node corresponding to the target node and obtaining the next level node corresponding to the target node based on the graph structure corresponding to the initial quantization model; An initial node link is constructed based on the upper-level node, the target node, and the lower-level node to obtain the multiple initial node links.

4. The method according to any one of claims 1 to 3, characterized in that The quantization loss value includes an output reconstruction loss value and a rounding loss value; the quantization result includes a quantized output result and a quantized weight parameter; The determining of the quantization loss value based on the quantization result of each initial node link includes: Obtaining an unquantized output result and a quantized output result corresponding to the initial node link; and performing a first loss calculation on the unquantized output result and the quantized output result to obtain the output reconstruction loss value; Obtaining a quantized weight parameter corresponding to the initial node link; and performing inverse quantization processing on the quantized weight parameter to obtain an inverse quantized weight parameter; Performing a second loss calculation on the unquantized weight parameter and the dequantized weight parameter corresponding to the initial node link to obtain the rounded loss value; The quantization loss value is determined based on the output reconstruction loss value and the rounding loss value.

5. The method according to any one of claims 1 to 4, characterized in that Determining a target node link based on the quantized loss value corresponding to each of the initial node links; After determining the preset quantization bit width corresponding to the target node link as the target bit width, the method further includes: Based on the target bit width, the quantization bit widths corresponding to the initial node links except the target node link are updated.

6. The method according to claim 5, characterized in that The initial quantization model includes multiple quantization layers; after updating the quantization bit widths corresponding to the initial node links other than the target node link based on the target bit width, the method further includes: Based on the connection order of each quantization layer in the initial quantization model, the initial quantization layer is re-determined; and the target bit width corresponding to the re-determined initial quantization layer is determined until the target bit width corresponding to all quantization layers included in the initial quantization model is determined.

7. The method according to any one of claims 1 to 6, characterized in that The parameter file also includes the number of tuning iterations; the method further includes: After constructing the target quantization model based on the target bit width, the target quantization model is iteratively optimized based on the number of tuning iterations to obtain an optimized target quantization model; wherein the optimized target quantization model is used for quantization processing.

8. A mixed precision quantization device, characterized in that: include an acquisition unit, configured to acquire a plurality of parameter files; wherein the parameter files include at least a preset quantization bit width and a preset number of iterations; the quantization bit width represents a bit width to which a parameter in a quantization layer can be quantized; the number of iterations includes the number of iterations used to determine the preset quantization bit width; and the preset quantization bit widths included in different parameter files are at least partially different; A generating unit, configured to generate an initial quantization layer based on each of the parameter files, to obtain a plurality of initial quantization layers; Wherein, the initial quantization layer represents the quantization layer of the graph model type; A construction unit is configured to construct an initial node link based on a target node included in each of the initial quantization layers to obtain a plurality of initial node links; and determine a quantization loss value based on a quantization result of each of the initial node links; wherein the target node represents a node including a convolution operator; A determination unit is used to determine a target node link based on the quantization loss value corresponding to each of the initial node links; and determine the preset quantization bit width corresponding to the target node link as the target bit width; wherein the quantization loss value corresponding to the target node link is the smallest; the target bit width is used to determine a target quantization layer; the target quantization layer is used to construct a target quantization model; the target quantization model represents a model for implementing mixed precision quantization.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the mixed-precision quantization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the mixed-precision quantization method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the mixed precision quantization method according to any one of claims 1 to 7 when executed by a processor.

Images