Task processing method and device based on model quantification, equipment and storage medium
By optimizing the quantization coefficient and rounding direction of the optimization unit in the Transformer model, the performance loss problem in the traditional model quantization solution is solved, and the accuracy of the quantitative model is improved.
Patent Information
- Application Number
- CN202411708331.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-27
- Publication Date
- 2025-03-18
AI Technical Summary
In traditional model quantization schemes, only optimization is performed for the quantization coefficient, resulting in performance losses of the quantization model and affecting the accuracy of task processing.
By updating the weight quantization coefficients and activation quantization coefficients of the optimization units in the Transformer model, the weight quantization rounding direction is optimized, and the difference between the output characteristics of the quantization model and the floating-point model is reduced.
It effectively improves the performance of the quantitative model and improves the task processing accuracy when using the quantitative model for task processing.
Smart Images
Figure CN119647530A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a task processing method, device, equipment and storage medium based on model quantization. Background Art
[0002] When a neural network (NN) is trained or forward inference is performed on the device side, a large number of floating-point float32 (32-bit floating-point number) multiply-accumulate operations (MACs) are required, which consumes storage and computing resources.
[0003] Using lower-precision data formats (such as low-precision floating-point numbers float16, float8, low-precision integer numbers in8, int4, etc.) instead of 32-bit floating-point operations can reduce resource consumption and improve model inference speed.
[0004] In traditional model quantization schemes, only the quantization coefficients are optimized, and the resulting quantization model usually suffers from a large performance loss, which in turn affects the accuracy of task processing using the quantization model. Summary of the invention
[0005] In view of this, the present application provides a task processing method, apparatus, device and storage medium based on model quantization.
[0006] Specifically, the present application is implemented through the following technical solutions:
[0007] According to a first aspect of an embodiment of the present application, a task processing method based on model quantization is provided, comprising:
[0008] For any optimization unit in the Transformer model, according to the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, the weight quantization coefficient and the activation quantization coefficient of the optimization unit are updated to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit; wherein, the first quantized output is the output of the optimization unit when the weight parameter of the optimization unit is quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0009] Under the condition that the floating-point weight parameter of the optimization unit and the target weight quantization coefficient and the target activation quantization coefficient are fixed, the weight quantization increment of the optimization unit is updated according to the difference between the second quantized output and the second floating-point output of the optimization unit with the goal of minimizing the difference, so as to obtain the target weight quantization increment of the optimization unit; wherein the second quantized output is the quantization output of the optimization unit when the weight parameter of the optimization unit is quantized according to the target quantization coefficient and the quantization adjustment is performed according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient, and the second floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0010] Quantizing the weight parameter of the optimization unit according to the target quantization coefficient, and determining the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, to obtain the target quantization weight parameter of the optimization unit;
[0011] When the Transformer model is used for task processing, for any optimization unit, forward reasoning calculation is performed on the input data of the optimization unit according to the target quantization weight parameter of the optimization unit, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient of the optimization unit.
[0012] According to a second aspect of an embodiment of the present application, a task processing device based on model quantization is provided, including:
[0013] A first determination unit is used for updating the weight quantization coefficient and activation quantization coefficient of any optimization unit in the Transformer model according to the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain a target weight quantization coefficient and a target activation quantization coefficient of the optimization unit; wherein the first quantized output is the output of the optimization unit when the weight parameter of the optimization unit is quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0014] A second determination unit is used to update the weight quantization increment of the optimization unit with the goal of minimizing the difference between the second quantized output and the second floating-point output of the optimization unit, while fixing the floating-point weight parameter of the optimization unit and the target weight quantization coefficient and the target activation quantization coefficient, so as to obtain the target weight quantization increment of the optimization unit; wherein the second quantized output is the quantization of the weight parameter of the optimization unit according to the target quantization coefficient, and the quantization adjustment is performed according to the weight quantization increment, and the quantization output of the optimization unit when the input / output of the optimization unit is quantized according to the target activation quantization coefficient, and the second floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0015] A quantization unit, used to quantize the weight parameter of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit;
[0016] A task processing unit is used to, when using the Transformer model for task processing, for any optimization unit, perform forward reasoning calculations on the input data of the optimization unit according to the target quantization weight parameters of the optimization unit, and quantize the input / output of the optimization unit according to the target activation quantization coefficient of the optimization unit.
[0017] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.
[0018] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, wherein machine executable instructions are stored in the storage medium, and when the machine executable instructions are executed by a processor, the method provided in the first aspect is implemented.
[0019] The technical solution provided by this application can at least bring the following beneficial effects:
[0020] In the process of quantizing the Transformer model, not only the quantization coefficients (including weight quantization coefficients and activation quantization coefficients) are optimized, but also the weight quantization rounding direction is optimized, which effectively improves the performance of the quantization model, and then improves the accuracy of task processing when using the quantization model for task processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1is a flowchart of a task processing method based on model quantization shown in an exemplary embodiment of the present application;
[0022] Figure 2 It is a schematic diagram of the main framework of artificial intelligence;
[0023] Figure 3 is a schematic diagram of a flow chart of optimizing quantization coefficients shown in an exemplary embodiment of the present application;
[0024] Figure 4 is a structural schematic diagram of a task processing device based on model quantization shown in an exemplary embodiment of the present application;
[0025] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0027] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0028] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0029] See also Figure 1 , is a flowchart of a task processing method based on model quantization provided in an embodiment of the present application, such as Figure 1 As shown, the task processing method based on model quantization may include the following steps:
[0030] Step S100: For any optimization unit in the Transformer model, based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, the weight quantization coefficient and the activation quantization coefficient of the optimization unit are updated to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit; wherein, the first quantized output is the quantized output of the optimization unit when the weight parameter of the optimization unit is quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input.
[0031] Step S110, under the condition that the floating-point weight parameters, target quantization coefficient and target activation quantization coefficient of the optimization unit are fixed, based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, the weight quantization increment of the optimization unit is updated to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the weight parameter of the optimization unit quantized according to the target quantization coefficient, and quantization adjustment is performed according to the weight quantization increment, and the quantization output of the optimization unit when the input / output of the optimization unit is quantized according to the target activation quantization coefficient, and the second floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input.
[0032] In the embodiment of the present application, considering that in the process of quantizing the Transformer model, the quantization coefficient finally used will affect the performance of the quantization model, it is necessary to determine a suitable quantization coefficient to reduce the difference in output characteristics between the quantization model and the floating-point model and improve the performance of the quantization model.
[0033] In addition, considering that in traditional model quantization schemes, rounding (Round-To-Nearest) is generally used for quantization, but not all quantization processes rounding is the optimal quantization rounding method, therefore, adaptive quantization rounding direction (such as rounding up or rounding down) can reduce the difference between the output characteristics of the quantization model and the floating-point model, thereby improving the performance of the quantization model.
[0034] That is, in the embodiment of the present application, the performance of the Transformer quantization model is improved by optimizing the quantization coefficient and the quantization rounding direction.
[0035] In the embodiment of the present application, the Transformer model can be regarded as modules (which can be called Blocks) stacked in a certain order.
[0036] Illustratively, a Block may include a functional unit consisting of a single layer or multiple layers.
[0037] In one example, the optimization unit in the Transformer model includes: Transformer Stage, Transformer Block, or a single linear layer.
[0038] For example, considering the mutual influence between layers in the Transformer model, if the quantization coefficients and quantization rounding directions of the multiple layers are optimized separately without considering the mutual influence within the multiple layer brackets, the optimization performance may be poor.
[0039] Accordingly, in order to take into account the mutual influence between layers in the process of optimizing the quantization coefficient and the quantization rounding direction, when determining the optimization unit in the Transformer model, the Transformer Stage or TransformerBlock can be used as the basic optimization unit. In addition to the Transformer Stage and Transformer Block, a single linear layer can also be used as the optimization unit.
[0040] It should be noted that, for a single nonlinear layer, since it does not have weight parameters and does not involve quantization of weight parameters, it is not necessary to optimize the single nonlinear layer.
[0041] In an embodiment of the present application, in the process of quantizing the Transformer model, Block can be used as the basic unit (which can be called an optimization unit), and the output difference of the optimization unit before and after quantization can be minimized, and the quantization coefficient and quantization rounding direction of each optimization unit can be optimized in sequence.
[0042] Exemplarily, the quantization coefficients may include weight quantization coefficients (used to quantize weight parameters) and activation quantization coefficients.
[0043] The activation quantization coefficient may include an input / output quantization coefficient used to quantize input data / output data.
[0044] In an embodiment of the present application, for any optimization unit in the Transformer model, the weight parameters of the optimization unit can be quantized according to the current weight quantization coefficient, and the input / output of the optimization unit can be quantized according to the current activation quantization coefficient. The output of the optimization unit (referred to as the first quantized output in this article) and the floating-point output of the optimization unit (referred to as the first floating-point output in this article) can be obtained respectively. Based on the difference between the first quantized output and the second floating-point output, with the goal of optimizing the difference, the weight quantization coefficient and activation quantization coefficient of the optimization unit are updated to obtain the weight quantization coefficient (referred to as the target weight quantization coefficient in this article) and activation quantization coefficient (referred to as the target activation quantization coefficient in this article) used by the optimization unit in the actual quantization process.
[0045] Exemplarily, for any optimization unit, in the process of updating the weight quantization coefficient and the activation quantization coefficient of the optimization unit, before the first update is completed, the values of the current weight quantization coefficient and the current activation quantization coefficient can be the initial values; after the first update is completed, the values of the current weight quantization coefficient and the current activation quantization can be the values of the weight quantization coefficient and the activation quantization coefficient when the last update was completed.
[0046] Exemplarily, for any optimization unit, when obtaining the first floating-point output of the optimization unit, each optimization unit in the Transformer model is in floating-point mode; when obtaining the first quantized output of the optimization unit, the optimization units before the optimization unit are in quantized mode.
[0047] It should be noted that, in the embodiment of the present application, when determining the difference between the first quantized output and the first floating-point output, it is the difference between the first quantized output and the first floating-point output of the optimization unit when the same data is input into the Transformer model.
[0048] For example, the prepared data (such as unlabeled images) can be input into the Transformer model in batches. When any batch of data is input into the Transformer model, for any optimization unit, the weight quantization coefficient and activation quantization coefficient of the optimization unit can be updated based on the difference between the first floating-point output and the first quantized output corresponding to the batch of data.
[0049] Exemplarily, for a non-first optimization unit or a non-last optimization unit, the activated quantization coefficient of the optimization unit may include a quantization coefficient for input data or a quantization coefficient for output.
[0050] When the activated quantization coefficient of the optimization unit is the quantization coefficient for the input data, for the last optimization unit, the activated quantization coefficient of the optimization unit may also include the quantization coefficient for the output data (the activated quantization coefficient of the first optimization unit is also the quantization coefficient for the input data).
[0051] When the activated quantization coefficient of the optimization unit is the quantization coefficient for the output data, for the first optimization unit, the activated quantization coefficient of the optimization unit may also include the quantization coefficient for the input data (the activated quantization coefficient of the last optimization unit is also the quantization coefficient for the output data).
[0052] In an embodiment of the present application, in order to further optimize the performance of the quantization model, after determining the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit, the quantization rounding direction can also be optimized to determine the optimal quantization rounding direction, rather than performing quantization rounding in a fixed rounding manner.
[0053] In addition, considering that the input / output of any optimization unit will change with the input of the Transformer model, and the input of the Transformer model is also unpredictable during actual task processing, it is impossible to determine the optimal quantization and rounding direction of the input / output of the optimization unit through testing.
[0054] Correspondingly, for any optimization unit, when the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit are determined, the weight quantization rounding direction can also be optimized.
[0055] Exemplarily, for any optimization unit, the floating-point weight parameters, target weight quantization coefficients, and target activation quantization coefficients of the optimization unit can be fixed, and the quantized output and floating-point output of the optimization unit can be obtained respectively when the input data of the Transformer model is the same.
[0056] Exemplarily, when obtaining the quantized output of the optimization unit, the weight parameters of the optimization unit can be quantized according to the above-mentioned target quantization coefficient, and the quantized weight parameters can be quantitatively adjusted according to the current weight quantization increment, and the input / output of the optimization unit can be quantized according to the above-mentioned target activation quantization coefficient, and the output of the optimization unit in this case (referred to as the second quantized output herein) can be obtained.
[0057] Exemplarily, for any optimization unit, the weight quantization increment of the optimization unit is used to adjust the quantization weight parameter of the optimization unit (the quantization weight parameter obtained by quantizing the weight parameter according to the target weight quantization coefficient), giving the quantization weight parameter an increment in the range of 0 to 1, so as to determine the optimal quantization rounding direction (rounding up or rounding down) when quantizing the weight parameter.
[0058] For example, if the difference between the second quantized output and the second floating-point output is smaller when the increment of the quantized weight parameter according to the weight quantization increment is closer to 1, it indicates that when the weight parameter is quantized, the performance of rounding up is better; if the difference between the second quantized output and the second floating-point output is smaller when the increment of the quantized weight parameter according to the weight quantization increment is closer to 0, it indicates that when the weight parameter is quantized, the performance of rounding down is better.
[0059] Exemplarily, when obtaining the floating-point output of the optimization unit, the output of the optimization unit (referred to herein as the second floating-point output) can be obtained when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input.
[0060] Exemplarily, for any optimization unit, during the process of updating the weight quantization increment of the optimization unit, before the first update is completed, the current weight quantization increment can be the initial value; after the first update is completed, the current weight quantization increment value can be the value of the weight quantization increment when the last update was completed.
[0061] It should be noted that, for any optimization unit, when the input of the Transformer model is the same, the first floating-point output and the second floating-point output are the same.
[0062] Exemplarily, based on the difference between the second quantized output and the second floating-point output of the optimization unit, the weight quantization increment of the optimization unit can be iteratively updated with the goal of minimizing the difference to obtain a final weight quantization increment of the optimization unit (referred to herein as target weight enhancement). The target weight quantization increment can be used to determine the weight quantization rounding direction of the optimization unit.
[0063] Step S120, quantize the weight parameter of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit.
[0064] In an embodiment of the present application, for any optimization unit, when the target weight quantization coefficient and target weight quantization increment of the optimization unit are determined in the above manner, the weight parameter of the optimization unit can be quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit can be determined according to the target weight quantization increment to obtain the final quantization weight parameter of the optimization unit (which can be called the target quantization weight parameter).
[0065] Step S130: When the Transformer model is used for task processing, for any optimization unit, forward reasoning calculation is performed on the input data of the optimization unit according to the target quantization weight parameter of the optimization unit, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient of the optimization unit.
[0066] In an embodiment of the present application, when the Transformer model is used for task processing, forward reasoning calculations can be performed on the input data of the optimization unit based on the target quantization weight parameters of each optimization unit, and the input / output of the optimization unit can be quantized based on the target activation quantization coefficient of the optimization unit.
[0067] Exemplarily, for any optimization unit, if the target activation quantization coefficient of the optimization unit includes a quantization coefficient for the input data, the optimization unit can quantize the input data of this optimization unit according to the quantization coefficient for the input data, and perform forward reasoning calculations on the quantized input data according to the target quantization weight parameters of this optimization unit, thereby improving the model reasoning speed while ensuring the model performance, thereby improving the task processing efficiency.
[0068] Exemplarily, the above-mentioned task processing includes but is not limited to NLP (Natural Language Processing) tasks (such as machine translation), or Speech (voice) tasks (such as speech recognition), or CV (Computer Vision) tasks (such as classification, target detection or target tracking tasks, etc.), etc.
[0069] It can be seen that in Figure 1 In the method flow shown, during the quantization of the Transformer model, not only the quantization coefficients (including weight quantization coefficients and activation quantization coefficients) are optimized, but also the weight quantization rounding direction is optimized, which effectively improves the performance of the quantization model, and then improves the accuracy of task processing when using the quantization model for task processing.
[0070] In some embodiments, the weight quantization coefficient and activation quantization coefficient of the optimization unit are updated based on the difference between the first quantized output and the first floating-point output of the optimization unit with the goal of minimizing the difference to obtain the target weight quantization coefficient and target activation quantization coefficient of the optimization unit, which may include:
[0071] Determine a first quantization loss of the optimization unit using a first preset loss function according to the first quantization output and the first floating-point output of the optimization unit;
[0072] According to the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the weight quantization coefficient and activation quantization coefficient of the optimization unit are iteratively updated using the gradient descent algorithm until the preset end condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.
[0073] Exemplarily, for any optimization unit, in the process of determining the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit, the quantization loss (which may be referred to as the first quantization loss) of the optimization unit can be determined based on the first quantization output and the first floating-point output of the optimization unit using a preset loss function (referred to as the first preset loss function in this article), such as a mean square error loss function, an absolute value loss function, etc., and based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the weight quantization coefficient and the activation quantization coefficient of the optimization unit are iteratively updated using a gradient descent algorithm until a preset end condition is reached, such as the number of iterations reaching a preset maximum number of iterations, and / or the loss function converges, etc., to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit.
[0074] In some embodiments, the above-mentioned updating the weight quantization increment of the optimization unit based on the difference between the second quantized output and the second floating-point output of the optimization unit with the goal of minimizing the difference to obtain the target weight quantization increment of the optimization unit may include:
[0075] Determine a second quantization loss of the optimization unit using a second preset loss function according to the second quantization output and the second floating-point output of the optimization unit;
[0076] According to the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the weight quantization increment of the optimization unit is iteratively updated using a gradient descent algorithm until a preset end condition is reached, thereby obtaining the target weight quantization increment of the optimization unit.
[0077] Exemplarily, for any optimization unit, when the target weight quantization coefficient of the optimization unit is determined, the second quantization output and the second floating-point output of the optimization unit can be determined in the above manner, and based on the second quantization output and the second floating-point output of the optimization unit, a preset loss function (referred to as the second preset loss function herein), such as a mean square error loss function, an absolute value loss function, etc., is used to determine the quantization loss of the optimization unit (referred to as the second quantization loss herein), and based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the weight quantization increment of the optimization unit is iteratively updated using a gradient descent algorithm until a preset end condition is reached, such as the number of iterations reaching a preset maximum number of iterations, and / or the loss function converges, etc., to obtain the target weight quantization increment of the optimization unit.
[0078] In one example, the weight parameter of the optimization unit is quantized according to the target weight quantization coefficient, and the quantization adjustment is performed according to the weight quantization increment, which can be achieved by the following formula:
[0079] W″=Floor(W / s)+sigmoid(V / t)
[0080] Wherein, W″ is the weight parameter after quantization adjustment of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, t is the hyperparameter for controlling the sigmoid function, and in the process of iteratively updating the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is a rounding-down operation.
[0081] Exemplarily, for any optimization unit, in the process of determining the optimal weight quantization rounding direction of the optimization unit, the quantization rounding function can be replaced by the Round function (i.e., rounding function) with the Floor function (i.e., rounding down function), and the weight parameters of the optimization unit are quantized according to the target weight quantization coefficient of the optimization unit, and the increment of the quantization weight parameters is determined using the sigmoid function based on the current weight quantization increment.
[0082] Among them, the sigmoid function is a common curve function with a monotonic property, which can map variables to between 0 and 1. When the variable is less than 0, the value of the sigmoid function is less than 0.5; when the variable is greater than 0, the value of the sigmoid function is greater than 0.5.
[0083] Through the above formula, when the value of V is greater than 0, an increment greater than 0.5 can be added to the quantization weight parameter, and the larger the value of V / t is, the closer the increment is to 1; when the value of V is less than 0, an increment less than 0.5 can be added to the quantization weight parameter, and the smaller the value of V / t is, the closer the increment is to 0. Furthermore, by iteratively updating V, the optimal weight quantization increment (i.e., the target weight quantization increment) can be determined, and the optimal weight quantization rounding direction of the optimization unit can be determined based on the target weight quantization increment.
[0084] In some embodiments, the weight parameter of the optimization unit is quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit, which can be achieved by the following formula:
[0085] W q =Floor(W / s)+(V>0?1:0)
[0086] Among them, W q is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, Floor() is the rounding-down operation, (V>0?1:0) means that when V>0 is true, the value is 1, and when V>0 is not true, the value is 0.
[0087] Exemplarily, when the target weight quantization increment is determined in the above manner, the optimal weight quantization rounding direction can be determined according to the target weight increment.
[0088] Exemplarily, when the target weight quantization increment is greater than 0, the optimal weight quantization rounding direction can be determined to be rounding up; when the target weight quantization increment is less than 0, the optimal weight quantization rounding direction can be determined to be rounding down.
[0089] In some embodiments, according to the quantized output and the floating point output of the optimization unit, the quantized loss of the optimization unit is determined by using a preset loss function, including:
[0090] Determining a standard deviation of the floating-point output of the optimization unit according to the floating-point output of the optimization unit;
[0091] The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating point output of the optimization unit is divided by the standard deviation to obtain the processed floating point output;
[0092] According to the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using a mean square error loss function.
[0093] For example, considering that the output features of the optimization unit are usually multi-dimensional data, and the differences in the values of different dimensions may be relatively large, for example, taking three-dimensional data as an example, the value of the first dimension may be 0-1, the value of the second dimension may be 10-100, and the value of the third dimension may be 100-1000. In this case, if the quantization loss is calculated directly based on the floating-point output and quantized output of the optimization unit using the mean square error loss function, the calculation result will focus too much on the value of a certain dimension, affecting the optimization effect.
[0094] In order to avoid the above problems and improve the optimization effect, in the process of determining the quantization loss of the optimization unit based on the quantization output and floating-point output of the optimization unit, for example, determining the first quantization loss of the optimization unit based on the first quantization output and the first floating-point output of the optimization unit, or determining the second quantization loss of the optimization unit based on the second quantization output and the second floating-point output of the optimization unit, you can first determine the standard deviation of the floating-point output of the optimization unit along a certain dimension based on the floating-point output of the optimization unit.
[0095] Take determining the first quantization loss of the optimization unit according to the first quantization output and the first floating-point output of the optimization unit as an example (the implementation of determining the second quantization loss of the optimization unit according to the second quantization output and the second floating-point output of the optimization unit can be obtained similarly).
[0096] For any optimization unit, the standard deviation of the first floating-point output of the optimization unit along a certain dimension can be determined based on the first floating-point output of the optimization unit, and the first quantized output of the optimization unit can be divided by the standard deviation to obtain the processed first quantized output, and the first floating-point output of the optimization unit can be divided by the standard deviation to obtain the processed first floating-point output. Then, based on the processed first quantized output and the processed first floating-point output, the first quantization loss of the optimization unit can be determined using the mean square error loss function.
[0097] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below in conjunction with specific embodiments.
[0098] 1. Below is a brief description of the main framework of artificial intelligence.
[0099] like Figure 2 As shown in the figure, the main framework of artificial intelligence can include five levels: application layer, algorithm layer, system layer, dependency layer, and device layer. There is a mutual dependency relationship from top to bottom. At the same time, it is biased towards practical applications upward and towards the underlying hardware downward.
[0100] 1.1. Application layer: By analyzing the needs, locate the problem to the corresponding branch of artificial intelligence.
[0101] 1.2. Algorithm layer: Design the model training strategy, loss function, and subsequent compression algorithm based on the application scenario.
[0102] 1.3. System layer: Use the deep learning framework to build models, train the built models, and perform computational graph analysis and model compression.
[0103] 1.4. Dependency layer: The language or deep learning framework that implements the algorithm uses the device's external interface and protocol to call the corresponding device.
[0104] 1.5. Device layer: It is composed of computing units and provides computing power support for artificial intelligence systems.
[0105] 2. The following describes the implementation of the Transformer model untrained quantization scheme in this embodiment (the data used in the Transformer model quantization process is unlabeled data).
[0106] In this embodiment, considering that the fixed rounding method used in the quantization process is not optimal, therefore, in the quantization process of the Transformer model, the difference in output characteristics between the quantization model and the floating-point model can be reduced by optimizing the weight quantization rounding direction and the quantization coefficient, thereby improving the performance of the quantization model.
[0107] 2.1 Quantifying Algorithm Performance
[0108] Model to be quantized: swin-tiny ImageNet classification model;
[0109] Precision setting: 848, input and output 8-bit quantization, weight 4-bit quantization (the floating-point model corresponds to 32 bits);
[0110] Data volume: 1024 unlabeled images;
[0111] Time required for quantification: about 30 minutes;
[0112] Exemplarily, the time required for quantization is independent of the quantization accuracy and depends on the number of algorithm iterations and the model size.
[0113] Performance: floating point performance acc@top1 = 81.18, quantized performance acc@top1 = 80.57.
[0114] 2.2. The basic process of the quantization scheme of this embodiment is as follows:
[0115] 2.2.1. Balance the numerical differences between channels of Shortcut addition through the Eltwise (element-level operation layer)-With-Bias (abbreviated as ELT) method;
[0116] For example, the Eltwise layer is a general term for a functional layer in a neural network, which is characterized by performing element-by-element (same position) calculations on two or more data blocks of the same size, and the calculation process can be addition or multiplication, etc.
[0117] During the data processing of the Eltwise layer, there may be large differences in the data accuracy of each data block input to the Eltwise layer, which in turn leads to large differences in the data range of each data block. The overall distribution variance of the element-level operation results obtained after performing element-level operations on each data block is also large, resulting in lower data accuracy for element-level operations.
[0118] In order to solve this problem, for each data block input to the Eltwise layer, a corresponding compensation coefficient can be set for each channel of each data block, that is, compensation coefficients refined to the input channel level are proposed. These compensation coefficients can compensate for the data range differences on each channel of each data block, and then compensate for the data range differences of these multiple data blocks, so that the data precision range of these multiple data blocks is also compensated, thereby converting these multiple data blocks into data blocks with the same data precision to align the data precision of data in different channels, so that the overall distribution variance of the element-level operation results obtained after element-level operations based on the compensated data is reduced, and the data precision is improved.
[0119] Exemplarily, after multiplying each channel of each data block by the corresponding compensation coefficient and adding the multiplication results, the summed result and the bias coefficient (which can be represented by bias) can be added.
[0120] Among them, the offset coefficient refers to the compensation coefficient used to correct the data zero-point drift. By performing element-level operations on n compensated data blocks to obtain operation results, the operation results are added to the bias coefficient. This can correct the zero-point drift after the element-level operation, reduce the zero-point drift that may occur in each data channel, and further reduce the data error of the element-level operation.
[0121] 2.2.2. Insert quantization nodes to quantize weights or quantize activations (i.e. input / output).
[0122] Illustratively, the precision of the quantization nodes is optional.
[0123] 2.2.3. Use picture data (which may include part or all of the above 1024 pictures) to initialize the quantization coefficients of the quantization nodes.
[0124] Exemplarily, the calculation method of the initial value of the quantization coefficient may be:
[0125] alpha=xmax / nlevel
[0126] Among them, xmax is the quantization boundary and nlevel is the number of quantization levels.
[0127] Exemplarily, the quantization boundary depends on a boundary truncation method, such as max, percentile, OMSE, etc.
[0128] Exemplarily, the number of quantization levels depends on the number of quantization bits (ie, precision) b.
[0129] For example, the quantization level can be calculated as follows:
[0130] nlevel=2 b-1 -1
[0131] 2.2.4. Select optimization unit
[0132] For example, the model can be viewed as a stack of modules in a certain order. A module can be a single layer or a combination of multiple layers.
[0133] Exemplarily, the module can be called a Block. The optimization algorithm uses the Block as a basic unit (ie, an optimization unit), and reduces the difference in Block output features by optimizing the quantization coefficient and the weight quantization rounding direction, thereby improving the performance of the quantization model.
[0134] Exemplarily, the method for selecting the optimization unit in this embodiment is as follows:
[0135] 2.2.4.1. A single Transformer Block is used as a basic optimization unit.
[0136] 2.2.4.2. In addition to the Transformer Block, a single linear layer is used as an optimization unit.
[0137] 2.2.5. Optimizing Quantization Coefficients
[0138] For example, the Round function is used as the quantization rounding function, the quantization coefficient is regarded as a learnable parameter, and the difference between the floating-point output and the quantization output of the same optimization unit is minimized to adjust the quantization coefficient (including the weight quantization coefficient and the activation quantization coefficient). The implementation process can be as follows: Figure 3 As shown, the following process is included:
[0139] 2.2.5.1. In floating point mode, input image data (such as the 1024 images above) (corresponding to Figure 3 Input: Float) in the optimization unit to obtain the floating-point output Y f1 (i.e. the first floating point output above) (corresponding to Figure 3Output: Float in
[0140] 2.2.5.2. In quantization mode, input image data (such as the above 1024 images) (corresponding to Figure 3 Input: Qunat in the optimization unit to obtain the quantitative output Y q1 (i.e. the first quantized output) (corresponding to Figure 3 Output in: Qunat);
[0141] 2.2.5.3. Calculate the optimization unit difference L(Y f1 , Y q1 )(i.e. the first quantization loss mentioned above)(corresponding to Figure 3 The quantization coefficients are iteratively updated through the gradient descent algorithm to obtain the final quantization coefficients (i.e., the target weight quantization coefficients and the target activation quantization coefficients mentioned above).
[0142] 2.2.6. Optimize the direction of weight quantization rounding
[0143] Exemplarily, the floating-point weight parameters and quantization coefficients (i.e., the above-mentioned target weight quantization coefficients and target activation quantization coefficients) can be fixed, and the weight quantization rounding direction can be optimized by optimizing the weight quantization increment. The specific implementation process is as follows:
[0144] 2.2.6.1. The quantization rounding function of the weight quantization node is replaced from the Round function to the Floor function;
[0145] 2.2.6.2. Define the weight quantization increment as parameter V and initialize the parameter;
[0146] 2.2.6.3. The quantization process in the iterative update process of V is:
[0147] W″=Floor(W / s)+sigmoid(V / t)
[0148] Wherein, W″ is the weight parameter after quantization adjustment of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, t is the hyperparameter for controlling the sigmoid function, and in the process of iteratively updating the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is a rounding-down operation.
[0149] Exemplarily, the iterative update process of V can be decomposed into the following execution steps:
[0150] 1) In floating point mode, input image data (such as the 1024 images mentioned above) and obtain the optimization unit floating point output Y f2 (ie the second floating point output mentioned above).
[0151] 2) In the quantization mode, input the image data (such as the 1024 images mentioned above) and obtain the quantized output Y of the optimization unit q2 (ie the second quantized output mentioned above).
[0152] 3) Calculate the optimization unit difference L(Y f2 , Y q2 ) (i.e., the second quantization loss mentioned above), V is iteratively updated through the gradient descent algorithm.
[0153] 4) During the iteration process, the temperature coefficient t decreases continuously, causing the Sigmoid function to degenerate into a step function.
[0154] In the process of quantizing the weight parameter according to the target weight quantization coefficient and the target weight quantization increment, the quantization process is:
[0155] W q =Floor(W / s)+(V>0?1:0)
[0156] Among them, W q is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, Floor() is the rounding-down operation, (V>0?1:0) means that when V>0 is true, the value is 1, and when V>0 is not true, the value is 0.
[0157] In this embodiment, it is considered that the output features of the optimization unit are usually multi-dimensional data, and the differences in the values of different dimensions may be relatively large. For example, taking three-dimensional data as an example, the value of the first dimension may be 0-1, the value of the second dimension may be 10-100, and the value of the third dimension may be 100-1000. In this case, if the quantization loss is calculated directly based on the floating-point output and quantized output of the optimization unit using the mean square error loss function, the calculation result will focus too much on the value of a certain dimension, affecting the optimization effect.
[0158] In order to avoid the above problems and improve the optimization effect, when determining the quantization loss of the optimization unit according to the quantization output and floating-point output of the optimization unit, for example, the above L(Y f1 , Y q1 ) or L(Y f2 , Y q2 ), the standard deviation of the Token dimension can be calculated using the floating-point output features to balance the Token differences between the floating-point output features and the quantized output features, and then the mean square error can be calculated.
[0159] For example, the standard deviation of the floating-point output of the optimization unit along a certain dimension can be determined based on the floating-point output of the optimization unit. Then, the quantized output and the floating-point output of the optimization unit are divided by the standard deviation to obtain the processed quantized output and the processed floating-point output, and the mean square error is calculated based on the processed quantized output and the processed floating-point output.
[0160] In this embodiment, the Transformer model quantized in the above manner can be used for natural language processing tasks, such as text similarity, text classification, machine translation, etc.; it can also be applied to speech tasks, such as speech recognition; it can also be used for visual tasks, such as image classification, target detection and target tracking.
[0161] The effects of the embodiments of the present application are described below with reference to specific application examples.
[0162] 1. License Plate Recognition Scenario
[0163] License plate recognition refers to the technology that can detect vehicles on the monitored road and automatically extract vehicle license plate information (including Chinese characters, English letters, Arabic numerals and license plate color) for processing. License plate recognition is one of the important components of modern intelligent transportation systems and is widely used. It is based on digital image processing, pattern recognition, computer vision and other technologies to analyze the vehicle images or video sequences taken by the camera to obtain the unique license plate number of each car, thereby completing the recognition process. Through some subsequent processing methods, parking fee management, traffic flow control index measurement, vehicle positioning, car anti-theft, highway speeding automatic supervision, red light electronic police, highway toll stations and other functions can be realized. The steps of license plate recognition are mainly to locate the license plate position in the picture first, then segment the characters in the license plate, and finally recognize the segmented characters to form the license plate number.
[0164] Among them, license plate character recognition is implemented based on a neural network. By adopting the Transformer model quantized by the method provided in this application for photographed character recognition, the recognition task can be completed quickly and efficiently.
[0165] 2. OCR text recognition scenario
[0166] OCR (Optical Character Recognition) text recognition refers to the process in which an electronic device (such as a scanner or digital camera) checks the characters printed on paper and then uses character recognition methods to translate the shapes into computer text; that is, the process of scanning text materials and then analyzing and processing the image files to obtain text and layout information.
[0167] Recognition speed is one of the main indicators for measuring the performance of an OCR system. By extracting text features using the quantized Transformer model provided by the method provided in this application, the recognition speed can be improved, thereby improving the practicality of the OCR product.
[0168] 3. Pedestrian Retrieval Scenario
[0169] Pedestrian retrieval is a technology that uses computer vision technology to determine whether there are specific pedestrians in an image or video sequence, which belongs to the problem of image retrieval.
[0170] Given a surveillance pedestrian image, retrieve the pedestrian image across devices. The core of pedestrian retrieval is how to find discriminative pedestrian expressions. Many recent methods use deep learning models to extract visual features. By using the quantized Transformer model of the method of this application, fast feature extraction can be achieved and time overhead can be reduced.
[0171] The method provided by the present application is described above. The device provided by the present application is described below:
[0172] See also Figure 4 , is a schematic diagram of a structure of a task processing device based on model quantization provided in an embodiment of the present application, such as Figure 4 As shown, the task processing device based on model quantization may include:
[0173] The first determination unit 410 is used for updating the weight quantization coefficient and activation quantization coefficient of any optimization unit in the Transformer model according to the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit; wherein the first quantized output is the output of the optimization unit when the weight parameter of the optimization unit is quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0174] A second determining unit 420 is used to update the weight quantization increment of the optimization unit with the goal of minimizing the difference between the second quantized output and the second floating-point output of the optimization unit, while fixing the floating-point weight parameter of the optimization unit and the target weight quantization coefficient and the target activation quantization coefficient, so as to obtain the target weight quantization increment of the optimization unit; wherein the second quantized output is the quantization of the weight parameter of the optimization unit according to the target quantization coefficient, and the quantization adjustment is performed according to the weight quantization increment, and the quantization output of the optimization unit when the input / output of the optimization unit is quantized according to the target activation quantization coefficient, and the second floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input;
[0175] A quantization unit 430 is used to quantize the weight parameter of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit;
[0176] The task processing unit 440 is used to perform forward reasoning calculations on the input data of any optimization unit according to the target quantization weight parameters of the optimization unit when the Transformer model is used for task processing, and to quantize the input / output of the optimization unit according to the target activation quantization coefficient of the optimization unit.
[0177] In some embodiments, the first determining unit 410 updates the weight quantization coefficient and the activation quantization coefficient of the optimization unit according to the difference between the first quantized output and the first floating-point output of the optimization unit with the goal of minimizing the difference, and obtains the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit, including:
[0178] Determine a first quantization loss of the optimization unit using a first preset loss function according to the first quantization output and the first floating-point output of the optimization unit;
[0179] According to the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the weight quantization coefficient and activation quantization coefficient of the optimization unit are iteratively updated using a gradient descent algorithm until a preset end condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.
[0180] In some embodiments, the second determining unit 420 updates the weight quantization increment of the optimization unit according to the difference between the second quantized output and the second floating-point output of the optimization unit with the goal of minimizing the difference to obtain the target weight quantization increment of the optimization unit, including:
[0181] Determine a second quantization loss of the optimization unit using a second preset loss function according to the second quantization output and the second floating-point output of the optimization unit;
[0182] According to the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the weight quantization increment of the optimization unit is iteratively updated using a gradient descent algorithm until a preset end condition is reached, thereby obtaining the target weight quantization increment of the optimization unit.
[0183] In some embodiments, the second determining unit 420 quantizes the weight parameter of the optimization unit according to the target weight quantization coefficient, and performs quantization adjustment according to the weight quantization increment, which is implemented by the following formula:
[0184]
[0185] Wherein, W″ is the weight parameter after quantization adjustment of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the weight quantization increment, t is the hyperparameter for controlling the sigmoid function, and in the process of iteratively updating the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is a rounding-down operation.
[0186] In some embodiments, the quantization unit 430 quantizes the weight parameter of the optimization unit according to the target quantization coefficient, and determines the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit, which is implemented by the following formula:
[0187] W q =Floor(W / s)+(V>0?1:0)
[0188] Among them, W q is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the target weight quantization increment, Floor() is the rounding-down operation, (V>0?1:0) means that when V>0 is true, the value is 1, and when V>0 is not true, the value is 0.
[0189] In some embodiments, the first determining unit 410 or the second determining unit 420 determines the quantization loss of the optimization unit according to the quantization output and the floating point output of the optimization unit using a preset loss function, including:
[0190] Determining a standard deviation of the floating-point output of the optimization unit according to the floating-point output of the optimization unit;
[0191] The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating point output of the optimization unit is divided by the standard deviation to obtain the processed floating point output;
[0192] According to the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using a mean square error loss function.
[0193] In some embodiments, the optimization unit in the Transformer model includes: a Transformer Stage, a Transformer Block, or a single linear layer.
[0194] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the domain adaptation-based task processing method described above.
[0195] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501 and a memory 502 storing machine executable instructions. The processor 501 and the memory 502 may communicate via a system bus 503. In addition, by reading and executing the machine executable instructions corresponding to the domain adaptation-based task processing logic in the memory 502, the processor 501 may execute the domain adaptation-based task processing method described above.
[0196] The memory 502 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.
[0197] In some embodiments, a storage medium is also provided, such as Figure 5 The memory 502 in the memory 502 is a machine-readable storage medium, and the storage medium stores machine-executable instructions. When the machine-executable instructions are executed by the processor, the task processing method based on domain adaptation described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0198] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0199] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A task processing method based on model quantization, characterized in that: include: For any optimization unit in the Transformer model, based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, the weight quantization coefficient and the activation quantization coefficient of the optimization unit are updated to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit; wherein the optimization unit in the Transformer model includes a Transformer Block; Under the condition that the floating-point weight parameter of the optimization unit and the target weight quantization coefficient and the target activation quantization coefficient are fixed, according to the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, the weight quantization increment of the optimization unit is updated to obtain the target weight quantization increment of the optimization unit; The weight parameter of the optimization unit is quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit.
2. The method according to claim 1, characterized in that The first quantized output is the output of the optimization unit when the weight parameter of the optimization unit is quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient; the first floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input; The second quantized output is the quantized output of the optimization unit when the weight parameter of the optimization unit is quantized according to the target quantization coefficient and quantized according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient. The second floating-point output is the floating-point output of the optimization unit when the weight parameter of the optimization unit is a floating-point weight parameter and the input is a floating-point input.
3. The method according to claim 1, characterized in that For any optimization unit in the Transformer model, according to the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, before updating the weight quantization coefficient and the activation quantization coefficient of the optimization unit, the method further includes: For the data blocks input by the Shortcut addition layer, each channel of each data block is multiplied by the corresponding compensation coefficient, and the multiplication result is added to the bias coefficient.
4. The method according to claim 1, characterized in that The step of updating the weight quantization increment of the optimization unit based on the difference between the second quantized output and the second floating-point output of the optimization unit with the goal of minimizing the difference to obtain the target weight quantization increment of the optimization unit includes: Determine a second quantization loss of the optimization unit using a second preset loss function according to the second quantization output and the second floating-point output of the optimization unit; According to the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the weight quantization increment of the optimization unit is iteratively updated using a gradient descent algorithm until a preset end condition is reached, thereby obtaining the target weight quantization increment of the optimization unit.
5. The method according to claim 4, characterized in that The weight parameter of the optimization unit is quantized according to the target weight quantization coefficient, and the quantization adjustment is performed according to the weight quantization increment, which is achieved by the following formula: W″=Floor(W / s)+sigmoid(V / t) Wherein, W″ is the weight parameter after quantization adjustment of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the weight quantization increment, t is the hyperparameter for controlling the sigmoid function, and in the process of iteratively updating the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is a rounding-down operation.
6. The method according to claim 1, characterized in that The weight parameter of the optimization unit is quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit, which is implemented by the following formula: W q =Floor(W / s)+(V>0?1:0) Among them, W q is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the target weight quantization increment, Floor() is the rounding-down operation, (V>0?1:0) means that when V>0 is true, the value is 1, and when V>0 is not true, the value is 0.
7. The method according to claim 1, characterized in that The method further comprises: When the Transformer model is used for task processing, for any optimization unit, forward reasoning calculation is performed on the input data of the optimization unit according to the target quantization weight parameter of the optimization unit, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient of the optimization unit.
8. A task processing device based on model quantization, characterized in that: include: A first determination unit is used for updating a weight quantization coefficient and an activation quantization coefficient of any optimization unit in the Transformer model according to a difference between a first quantized output and a first floating-point output of the optimization unit with the goal of minimizing the difference, so as to obtain a target weight quantization coefficient and a target activation quantization coefficient of the optimization unit; wherein the optimization unit in the Transformer model includes a Transformer Block; A second determining unit is used to update the weight quantization increment of the optimization unit with the goal of minimizing the difference between the second quantized output and the second floating-point output of the optimization unit, while fixing the floating-point weight parameter of the optimization unit and the target weight quantization coefficient and the target activation quantization coefficient, so as to obtain a target weight quantization increment of the optimization unit; The quantization unit is used to quantize the weight parameter of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameter of the optimization unit.
9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores machine executable instructions, and when the machine executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.