Model-based quantization-based task processing methods, devices, equipment, and storage media

Abstract

This application provides a model-based quantization-based task processing method, apparatus, device, and storage medium. The model-based quantization-based task processing method includes: updating the weight quantization coefficients and activation quantization coefficients of an optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit in a Transformer model; updating the weight quantization increment of the optimization unit based on the difference between the second quantized output and the second floating-point output of the optimization unit; quantizing the weight parameters of the optimization unit based on the target quantization coefficients, and determining the weight quantization rounding direction of the optimization unit based on the target weight quantization increments; performing forward inference calculations on the input data of the optimization unit based on the target quantized weight parameters of the optimization unit, and quantizing the input / output of the optimization unit based on the target activation quantization coefficients of the optimization unit. This method can improve the accuracy of task processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a task processing method, apparatus, device and storage medium based on model quantization. Background Technology

[0002] When training a neural network (NN) or performing forward inference on a device, a large number of floating-point 32-bit multiply-accumulate operations (MACs) are required, consuming storage and computing resources.

[0003] Using lower-precision data formats (such as low-precision floating-point numbers float16, float8, low-precision integers in8, int4, etc.) instead of 32-bit floating-point numbers for operations can reduce resource consumption and improve model inference speed.

[0004] Traditional model quantization schemes only optimize the quantization coefficients, resulting in quantized models that often suffer from significant performance loss, which in turn affects the accuracy of task processing using the quantized models. Summary of the Invention

[0005] In view of this, this application provides a task processing method, apparatus, device and storage medium based on model quantization.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] According to a first aspect of the embodiments of this application, a task processing method based on model quantization is provided, comprising:

[0008] For any optimization unit in the Transformer model, based on the difference between the first quantized output and the first floating-point output of the optimization unit, the weight quantization coefficient and activation quantization coefficient of the optimization unit are updated with the goal of minimizing the difference, to obtain the target weight quantization coefficient and target activation quantization coefficient of the optimization unit; wherein, the first quantized output is the output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input;

[0009] With the floating-point weight parameters of the optimization unit fixed and the target weight quantization coefficient and target activation quantization coefficient fixed, the weight quantization increment of the optimization unit is updated based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target quantization coefficient and quantized according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient; the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input;

[0010] The weight parameters of the optimization unit are quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment, so as to obtain the target quantized weight parameters of the optimization unit.

[0011] When using the Transformer model for task processing, for any optimization unit, forward inference calculation is performed on the input data of the optimization unit according to the target quantization weight parameter of the optimization unit, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient of the optimization unit.

[0012] According to a second aspect of the embodiments of this application, a task processing apparatus based on model quantization is provided, comprising:

[0013] The first determining unit is configured to, for any optimization unit in the Transformer model, update the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit; wherein, the first quantization output is the output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficients and the input / output of the optimization unit is quantized according to the activation quantization coefficients, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input;

[0014] The second determining unit is configured to, with the floating-point weight parameters of the optimization unit fixed and the target weight quantization coefficient and target activation quantization coefficient fixed, update the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target quantization coefficient and quantized according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient; and the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input.

[0015] A quantization unit is used to quantize the weight parameters of the optimization unit according to the target quantization coefficient, and to determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, so as to obtain the target quantized weight parameters of the optimization unit.

[0016] The task processing unit is configured to, when performing task processing using the Transformer model, perform forward inference calculations on the input data of any optimization unit based on the target quantization weight parameters of that optimization unit, and quantize the input / output of that optimization unit based on the target activation quantization coefficients of that optimization unit.

[0017] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0018] According to a fourth aspect of the embodiments of this application, a storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0019] The technical solution provided in this application can bring at least the following beneficial effects:

[0020] In the process of quantizing the Transformer model, not only were the quantization coefficients (including weight quantization coefficients and activation quantization coefficients) optimized, but the rounding direction of weight quantization was also optimized, which effectively improved the performance of the quantization model and, consequently, improved the accuracy of task processing when using the quantization model for task processing. Attached Figure Description

[0021] Figure 1This is a flowchart illustrating a model-quantized task processing method according to an exemplary embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the main framework of artificial intelligence;

[0023] Figure 3 This is a schematic diagram illustrating an exemplary embodiment of the present application of optimizing quantization coefficients;

[0024] Figure 4 This is a schematic diagram illustrating the structure of a task processing device based on model quantization, as shown in an exemplary embodiment of this application.

[0025] Figure 5 This is a schematic diagram of the hardware structure of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0028] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0029] Please see Figure 1 This is a flowchart illustrating a task processing method based on model quantization provided in an embodiment of this application. Figure 1 As shown, this model-quantization-based task processing method may include the following steps:

[0030] Step S100: For any optimization unit in the Transformer model, based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, update the weight quantization coefficient and the activation quantization coefficient of the optimization unit to obtain the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit; wherein, the first quantization output is the quantized output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficient and the input / output of the optimization unit are quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input.

[0031] Step S110: With the floating-point weight parameters, target quantization coefficient, and target activation quantization coefficient of the optimization unit fixed, the weight quantization increment of the optimization unit is updated based on the difference between the second quantization output and the second floating-point output, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target quantization coefficient and adjusted according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient; the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input.

[0032] In this embodiment of the application, considering that the quantization coefficients used in the process of quantizing the Transformer model will affect the performance of the quantized model, it is necessary to determine appropriate quantization coefficients to reduce the difference between the output features of the quantized model and the floating-point model and improve the performance of the quantized model.

[0033] Furthermore, considering that traditional model quantization schemes generally use round-to-nearest rounding, but not all rounding is the optimal rounding method, adaptive rounding direction (such as rounding up or rounding down) can reduce the difference between the output features of the quantized model and the floating-point model, thereby improving the performance of the quantized model.

[0034] In this embodiment of the application, the performance of the Transformer quantization model is improved by optimizing the quantization coefficients and the quantization rounding direction.

[0035] In this embodiment of the application, the Transformer model can be viewed as modules (which can be called Blocks) stacked in a certain order.

[0036] For example, a block may include a single layer or a functional unit consisting of multiple layers.

[0037] In one example, the optimization units in a Transformer model include: a Transformer Stage, a Transformer Block, or a single linear layer.

[0038] For example, considering the interrelationships between layers in a Transformer model, optimizing the quantization coefficients and quantization rounding directions of each layer individually without considering the interrelationships within the scaffold of those layers may result in poor optimization performance.

[0039] Accordingly, in order to take into account the inter-layer interaction during the optimization of quantization coefficients and quantization rounding direction, when determining the optimization unit in the Transformer model, the Transformer Stage or Transformer Block can be used as the basic optimization unit. In addition to the Transformer Stage and Transformer Block, a single linear layer can be used as the optimization unit.

[0040] It should be noted that, for a single nonlinear layer, since it does not have weight parameters and does not involve the quantization of weight parameters, it is not necessary to optimize for a single nonlinear layer.

[0041] In this embodiment of the application, during the quantization process of the Transformer model, a Block can be used as the basic unit (which can be called an optimization unit). With the goal of minimizing the output difference of the optimization unit before and after quantization, the quantization coefficient and quantization rounding direction of each optimization unit are optimized sequentially.

[0042] For example, the quantization coefficients may include weight quantization coefficients (used to quantize weight parameters) and activation quantization coefficients.

[0043] The activation quantization coefficients can include input / output quantization coefficients, which are used to quantize input / output data.

[0044] In this embodiment, for any optimization unit in the Transformer model, the following can be obtained: the weight parameters of the optimization unit are quantized according to the current weight quantization coefficients, and the input / output of the optimization unit is quantized according to the current activation quantization coefficients. The output of the optimization unit (referred to as the first quantized output) is obtained when the weight parameters are floating-point weight parameters and the input is a floating-point input. Based on the difference between the first quantized output and the second floating-point output, with the goal of optimizing this difference, the weight quantization coefficients and activation quantization coefficients of the optimization unit are updated to obtain the weight quantization coefficients (referred to as the target weight quantization coefficients) and activation quantization coefficients (referred to as the target activation quantization coefficients) used by the optimization unit in the actual quantization process.

[0045] For example, for any optimization unit, during the process of updating the weight quantization coefficient and activation quantization coefficient of the optimization unit, before the first update is completed, the current weight quantization coefficient and the current activation quantization coefficient can be the initial values; after the first update is completed, the current weight quantization coefficient and the current activation quantization coefficient can be the values ​​of the weight quantization coefficient and the activation quantization coefficient when the last update was completed.

[0046] For example, when the first floating-point output of any optimization unit is obtained, all optimization units in the Transformer model are in floating-point mode; when the first quantized output of any optimization unit is obtained, all optimization units before this optimization unit are in quantized mode.

[0047] It should be noted that, in the embodiments of this application, when determining the difference between the first quantized output and the first floating-point output, it is the difference between the first quantized output and the first floating-point output of the optimization unit when the same data is input to the Transformer model.

[0048] For example, prepared data (such as unlabeled images) can be input into the Transformer model in batches. When any batch of data is input into the Transformer model, for any optimization unit, the weight quantization coefficient and activation quantization coefficient of the optimization unit can be updated based on the difference between the first floating-point output and the first quantization output corresponding to the data of that batch.

[0049] For example, for an optimization unit that is neither the first nor the last, the activation quantization coefficient of the optimization unit may include quantization coefficients for the input data or quantization coefficients for the output.

[0050] When the activation quantization coefficient of the optimization unit is a quantization coefficient for the input data, the activation quantization coefficient of the last optimization unit may also include a quantization coefficient for the output data (the activation quantization coefficient of the first optimization unit is also a quantization coefficient for the input data).

[0051] When the activation quantization coefficient of the optimization unit is a quantization coefficient for the output data, for the first optimization unit, the activation quantization coefficient of the optimization unit may also include a quantization coefficient for the input data (the activation quantization coefficient of the last optimization unit is also a quantization coefficient for the output data).

[0052] In this embodiment of the application, in order to further optimize the performance of the quantization model, after determining the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit, the quantization rounding direction can also be optimized to determine the optimal quantization rounding direction, instead of performing quantization rounding according to a fixed rounding method.

[0053] Furthermore, considering that the input / output of any optimization unit will change with the input of the Transformer model, and that the input of the Transformer model is also unpredictable in actual task processing, it is impossible to determine the optimal quantization and rounding direction of the input / output of the optimization unit through testing.

[0054] Accordingly, for any optimization unit, given that the target weight quantization coefficient and the target activation quantization coefficient of the optimization unit are determined, the weight quantization rounding direction can also be optimized.

[0055] For example, for any optimization unit, the floating-point weight parameter, the target weight quantization coefficient, and the target activation quantization coefficient of the optimization unit can be fixed, and the quantized output and floating-point output of the optimization unit can be obtained respectively when the input data of the Transformer model are the same.

[0056] For example, when obtaining the quantization output of the optimization unit, the weight parameters of the optimization unit can be quantized according to the target quantization coefficient mentioned above, and the quantized weight parameters can be quantized and adjusted according to the current weight quantization increment. The input / output of the optimization unit can be quantized according to the target activation quantization coefficient mentioned above, and the output of the optimization unit in this case (referred to as the second quantization output in this document) can be obtained.

[0057] For example, for any optimization unit, the weight quantization increment of the optimization unit is used to adjust the quantization weight parameter of the optimization unit (the quantization weight parameter obtained by quantizing the weight parameter according to the target weight quantization coefficient), giving the quantization weight parameter an increment with a value range of 0 to 1, so as to determine the optimal quantization rounding direction (rounding up or rounding down) when quantizing the weight parameter.

[0058] For example, if the increment of the quantization weight parameter based on the weight quantization increment is closer to 1, the difference between the second quantization output and the second floating-point output is smaller, indicating that rounding up performs better when quantizing the weight parameter; if the increment of the quantization weight parameter based on the weight quantization increment is closer to 0, the difference between the second quantization output and the second floating-point output is smaller, indicating that rounding down performs better when quantizing the weight parameter.

[0059] For example, when obtaining the floating-point output of the optimization unit, the output of the optimization unit (referred to as the second floating-point output in this document) can be obtained when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input.

[0060] For example, for any optimization unit, during the process of updating the weight quantization increment of the optimization unit, before the first update is completed, the current weight quantization increment can be the initial value; after the first update is completed, the current weight quantization increment can be the value of the weight quantization increment when the last update was completed.

[0061] It should be noted that for any optimization unit, with the same input to the Transformer model, the first floating-point output and the second floating-point output are the same.

[0062] For example, the weight quantization increment of the optimization unit can be iteratively updated based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the final weight quantization increment of the optimization unit (referred to as the target weight enhancement in this paper). The target weight quantization increment can be used to determine the weight quantization rounding direction of the optimization unit.

[0063] Step S120: Quantize the weight parameters of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment to obtain the target quantization weight parameters of the optimization unit.

[0064] In this embodiment of the application, for any optimization unit, after determining the target weight quantization coefficient and the target weight quantization increment of the optimization unit in the manner described above, the weight parameters of the optimization unit can be quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit can be determined according to the target weight quantization increment to obtain the final quantized weight parameters of the optimization unit (which can be called target quantized weight parameters).

[0065] Step S130: When using the Transformer model for task processing, for any optimization unit, perform forward inference calculation on the input data of the optimization unit according to the target quantization weight parameter of the optimization unit, and quantize the input / output of the optimization unit according to the target activation quantization coefficient of the optimization unit.

[0066] In this embodiment of the application, when using the Transformer model for task processing, the input data of the optimization unit can be forward-inferenced based on the target quantization weight parameters of each optimization unit, and the input / output of the optimization unit can be quantized based on the target activation quantization coefficient of the optimization unit.

[0067] For example, for any optimization unit, if the target activation quantization coefficient of the optimization unit includes a quantization coefficient for the input data, then the optimization unit can quantize the input data of the optimization unit according to the quantization coefficient for the input data, and perform forward inference calculation on the quantized input data according to the target quantization weight parameter of the optimization unit. While ensuring model performance, the model inference speed is improved, thereby improving task processing efficiency.

[0068] For example, the above-mentioned task processing includes, but is not limited to, NLP (Natural Language Processing) tasks (such as machine translation), or Speech tasks (such as speech recognition), or CV (Computer Vision) tasks (such as classification, object detection, or object tracking tasks).

[0069] It can be seen that, in Figure 1 In the method flow shown, during the quantization of the Transformer model, not only are the quantization coefficients (including weight quantization coefficients and activation quantization coefficients) optimized, but the rounding direction of weight quantization is also optimized, which effectively improves the performance of the quantization model and, consequently, improves the accuracy of task processing when using the quantization model for task processing.

[0070] In some embodiments, updating the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit, may include:

[0071] Based on the first quantization output and the first floating-point output of the optimization unit, the first quantization loss of the optimization unit is determined using the first preset loss function;

[0072] Based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization coefficient and activation quantization coefficient of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.

[0073] For example, for any optimization unit, in the process of determining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit, the quantization loss (which can be called the first quantization loss) of the optimization unit can be determined by using the first quantization output and the first floating-point output of the optimization unit, and using a preset loss function (referred to as the first preset loss function in this paper), such as the mean square error loss function, the absolute value loss function, etc. Based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization coefficient and activation quantization coefficient of the optimization unit until the preset termination condition is reached, such as the number of iterations reaching the preset maximum number of iterations, and / or the loss function converges, etc., to obtain the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.

[0074] In some embodiments, updating the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit, may include:

[0075] Based on the second quantization output and the second floating-point output of the optimization unit, the second quantization loss of the optimization unit is determined using the second preset loss function;

[0076] Based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization increment of the optimization unit until the preset termination condition is met, thus obtaining the target weight quantization increment of the optimization unit.

[0077] For example, for any optimization unit, given the target weight quantization coefficient of the optimization unit, the second quantization output and the second floating-point output of the optimization unit can be determined in the manner described above. Based on the second quantization output and the second floating-point output of the optimization unit, a preset loss function (referred to as the second preset loss function in this paper), such as the mean square error loss function or the absolute value loss function, is used to determine the quantization loss of the optimization unit (referred to as the second quantization loss in this paper). Based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization increment of the optimization unit until a preset termination condition is reached, such as the number of iterations reaching the preset maximum number of iterations, and / or the loss function converging, etc., to obtain the target weight quantization increment of the optimization unit.

[0078] In one example, the weight parameters of the optimization unit are quantized according to the target weight quantization coefficient, and the quantization is adjusted according to the weight quantization increment. This can be achieved using the following formula:

[0079]

[0080] Where W″ is the quantized weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, t is the hyperparameter controlling the sigmoid function, and t gradually decreases during the iterative update of the weight quantization increment of the optimization unit. Floor() is the floor function.

[0081] For example, for any optimization unit, in the process of determining the optimal weight quantization rounding direction of the optimization unit, the quantization rounding function can be replaced by the Round function (i.e., the rounding function) and replaced by the Floor function (i.e., the floor function). The weight parameters of the optimization unit are quantized according to the target weight quantization coefficient of the optimization unit, and the increment of the quantized weight parameters is determined by the sigmoid function according to the current weight quantization increment.

[0082] The sigmoid function is a common curve function with monotonically increasing properties, mapping variables to the range of 0 to 1. When the variable is less than 0, the sigmoid function takes a value less than 0.5; when the variable is greater than 0, the sigmoid function takes a value greater than 0.5.

[0083] Using the above formula, when V is greater than 0, an increment greater than 0.5 can be added to the quantization weight parameter, and the larger the value of V / t, the closer this increment is to 1; when V is less than 0, an increment less than 0.5 can be added to the quantization weight parameter, and the smaller the value of V / t, the closer this increment is to 0. Therefore, by iteratively updating V, the optimal weight quantization increment (i.e., the target weight quantization increment) can be determined, and the optimal weight quantization rounding direction of the optimization unit can be determined based on the target weight quantization increment.

[0084] In some embodiments, the weight parameters of the optimization unit are quantized according to the target quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment to obtain the target quantized weight parameters of the optimization unit. This can be achieved by the following formula:

[0085] W q =Floor(W / s) + (V>0? 1:0)

[0086] Among them, W q Here, is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, Floor() is the floor function, and (V>0 ? 1 ∶ 0) means that the value is 1 when V>0 is true, and the value is 0 when V>0 is false.

[0087] For example, if the target weight quantization increment is determined in the manner described above, the optimal weight quantization rounding direction can be determined based on the target weight increment.

[0088] For example, when the target weight quantization increment is greater than 0, the optimal weight quantization rounding direction can be determined to be rounding up; when the target weight quantization increment is less than 0, the optimal weight quantization rounding direction can be determined to be rounding down.

[0089] In some embodiments, the quantization loss of the optimization unit is determined using a preset loss function based on the quantization output and floating-point output of the optimization unit, including:

[0090] Based on the floating-point output of the optimization unit, determine the standard deviation of the floating-point output of the optimization unit;

[0091] The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating-point output of the optimization unit is divided by the standard deviation to obtain the processed floating-point output.

[0092] Based on the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using the mean square error loss function.

[0093] For example, considering that the output features of the optimization unit are usually multi-dimensional data, and the differences in the values ​​of different dimensions may be relatively large, for example, taking 3-dimensional data as an example, the value of the first dimension may be 0 to 1, the value of the second dimension may be 10 to 100, and the value of the third dimension may be 100 to 1000. In this case, if the quantization loss is calculated directly based on the floating-point output and quantization output of the optimization unit using the mean squared error loss function, the calculation result will focus too much on the value of a certain dimension, affecting the optimization effect.

[0094] To avoid the above problems and improve the optimization effect, when determining the quantization loss of the optimization unit based on its quantization output and floating-point output, for example, determining the first quantization loss of the optimization unit based on its first quantization output and first floating-point output, or determining the second quantization loss of the optimization unit based on its second quantization output and second floating-point output, the standard deviation of the floating-point output of the optimization unit along a certain dimension can be determined first based on the floating-point output of the optimization unit.

[0095] Taking the determination of the first quantization loss of the optimization unit based on the first quantization output and the first floating-point output of the optimization unit as an example (the implementation of determining the second quantization loss of the optimization unit based on the second quantization output and the second floating-point output of the optimization unit can be obtained in the same way).

[0096] For any optimization unit, the standard deviation of the first floating-point output of the optimization unit along a certain dimension can be determined based on the first floating-point output of the optimization unit. The first quantized output of the optimization unit is then divided by the standard deviation to obtain the processed first quantized output. The first floating-point output of the optimization unit is also divided by the standard deviation to obtain the processed first floating-point output. Furthermore, based on the processed first quantized output and the processed first floating-point output, the first quantization loss of the optimization unit is determined using the mean square error loss function.

[0097] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below in conjunction with specific embodiments.

[0098] 1. The following is a brief explanation of the main framework of artificial intelligence.

[0099] like Figure 2 As shown, the main framework of artificial intelligence can include five layers: application layer, algorithm layer, system layer, dependency layer, and device layer. These layers are interdependent from top to bottom. Furthermore, the upper layers lean towards practical applications, while the lower layers lean towards the underlying hardware.

[0100] 1.1 Application Layer: By analyzing the requirements, the problem is located in the corresponding branch of artificial intelligence.

[0101] 1.2 Algorithm Layer: Design the training strategy for the model, the loss function, and subsequent compression algorithms according to the application scenario.

[0102] 1.3 System Layer: Utilize deep learning frameworks to build models, train the built models, and perform computation graph analysis and model compression.

[0103] 1.4 Dependency Layer: The language or deep learning framework used to implement the algorithm, which calls the corresponding device using the device's external interface and protocol.

[0104] 1.5 Device Layer: Composed of computing units, providing computing power support for artificial intelligence systems.

[0105] 2. The implementation of the Transformer model training-free quantization scheme (the data used in the Transformer model quantization process is unlabeled data) in this embodiment will be described below.

[0106] In this embodiment, considering that using a fixed rounding method during quantization is not optimal, the difference between the output features of the quantized model and the floating-point model can be reduced by optimizing the weight quantization rounding direction and quantization coefficients during the quantization process of the Transformer model, thereby improving the performance of the quantized model.

[0107] 2.1 Quantization Algorithm Performance

[0108] Model to be quantized: Swin-tiny ImageNet classification model;

[0109] Precision setting: 848, input and output 8-bit quantization, weight 4-bit quantization (floating-point models correspond to 32 bits);

[0110] Data volume: 1024 unlabeled images;

[0111] Quantification time: Approximately 30 minutes;

[0112] For example, the time required for quantization is independent of the quantization accuracy, but depends on the number of algorithm iterations and the model size.

[0113] Performance: Floating-point performance acc@top1 = 81.18, quantization performance acc@top1 = 80.57.

[0114] 2.2 The basic process of the quantization scheme in this embodiment is as follows:

[0115] 2.2.1. Balance the numerical differences between channels in Shortcut addition using the Eltwise (element-level operation layer) - With-Bias (ELT) method;

[0116] For example, an Eltwise layer is a general term for a type of functional layer in a neural network. Its characteristic is that it performs element-wise (at the same position) calculations on two or more data blocks of the same size. The calculation process can be addition or multiplication, etc.

[0117] During data processing in the Eltwise layer, the precision of each data block input to the Eltwise layer may vary significantly, which in turn leads to significant differences in the data range of each data block. Consequently, the overall distribution variance of the element-level operation results obtained after performing element-level operations on each data block is also large, resulting in lower data precision of the element-level operations.

[0118] To address this issue, for each data block input to the Eltwise layer, a corresponding compensation coefficient can be set for each channel of each data block. This involves refining the compensation coefficients down to the input channel level. These compensation coefficients can compensate for the differences in data range across different channels of each data block, thereby compensating for the differences in data range across multiple data blocks. This also compensates for the data precision range of these multiple data blocks, transforming them into data blocks with the same data precision. This aligns the data precision of data from different channels, resulting in a reduction in the overall distribution variance of the element-level operation results obtained after performing element-level operations based on the compensated data, thus increasing the data precision.

[0119] For example, after multiplying each channel of each data block by the corresponding compensation coefficient and summing the results, the sum can be added to the bias coefficient (which can be represented by bias).

[0120] The offset coefficient is a compensation coefficient used to correct zero-point drift in data. After performing element-level operations on n compensation data blocks to obtain the operation results, the operation results are added to the offset coefficient to correct the zero-point drift after the element-level operation, reduce the possible zero-point drift in each data channel, and further reduce the data error of the element-level operation.

[0121] 2.2.2 Insert quantization nodes for quantization weights or quantization activations (i.e., inputs / outputs).

[0122] For example, the precision of the quantization node is optional.

[0123] 2.2.3. Initialize the quantization coefficients of the quantization nodes with image data (which may include some or all of the 1024 images mentioned above).

[0124] For example, the initial value of the quantization coefficient can be calculated as follows:

[0125] alpha = xmax / nlevel

[0126] Where xmax is the quantization boundary and nlevel is the number of quantization levels.

[0127] For example, the quantization boundary depends on the boundary truncation method. For instance, boundary truncation methods may include: max, percentile, OMSE, etc.

[0128] For example, the number of quantization levels depends on the number of quantization bits (i.e., precision) b.

[0129] For example, the level of quantification can be calculated in the following way:

[0130] nlevel=2 b-1 -1

[0131] 2.2.4 Selecting Optimization Units

[0132] For example, a model can be viewed as modules stacked in a certain order. A module can be a single layer or a combination of multiple layers.

[0133] For example, a module can be referred to as a Block. The optimization algorithm uses Block as the basic unit (i.e., the optimization unit), and improves the performance of the quantization model by optimizing the quantization coefficients and the weight quantization rounding direction to reduce the differences in Block output features.

[0134] For example, the method for selecting the optimization unit in this embodiment is as follows:

[0135] 2.2.4.1. A single Transformer Block is considered as a basic optimization unit;

[0136] 2.2.4.2 Except for the Transformer Block, a single linear layer is used as an optimization unit.

[0137] 2.2.5 Optimize Quantization Coefficient

[0138] For example, using the Round function as the quantization rounding function, the quantization coefficients are treated as learnable parameters. The quantization coefficients (including weighted quantization coefficients) are adjusted by minimizing the difference between the floating-point output and the quantized output of the same optimization unit. Figure 3 The implementation process can be as follows: (W and s) and activation quantization coefficients. Figure 3 As shown, the process includes the following:

[0139] 2.2.5.1. In floating-point mode, input image data (such as the 1024 images mentioned above) (corresponding to...) Figure 3 Input: Float), retrieve the floating-point output Y of the optimization unit. f1 (i.e., the first floating-point output mentioned above) (corresponding to) Figure 3 Output: Float;

[0140] 2.2.5.2 In quantization mode, input image data (such as the 1024 images mentioned above) (corresponding to...) Figure 3 Input: Qunet), obtain the quantized output Y of the optimization unit. q1 (i.e., the first quantization output mentioned above) (corresponding to) Figure 3 Output: Qunat);

[0141] 2.2.5.3 Calculate the difference L(Y) between optimized units f1 Y q1 (i.e., the first quantization loss mentioned above) (corresponding to) Figure 3 The quantization coefficients are iteratively updated using the LSQ algorithm (i.e., the target weight quantization coefficients and the target activation quantization coefficients mentioned above) to obtain the final quantization coefficients.

[0142] 2.2.6 Optimize the direction of weight quantization and rounding

[0143] For example, the floating-point weight parameters and quantization coefficients (i.e., the target weight quantization coefficient and the target activation quantization coefficient mentioned above) can be fixed, and the weight quantization rounding direction can be optimized by optimizing the weight quantization increment. The specific implementation process is as follows:

[0144] 2.2.6.1 The quantization rounding function of the weighted quantization node is replaced from the Round function to the Floor function;

[0145] 2.2.6.2 Define the weight quantization increment as parameter V, and initialize this parameter;

[0146] 2.2.6.3 The quantization process during the iterative update of V is as follows:

[0147]

[0148] Where W″ is the quantized weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, t is the hyperparameter controlling the sigmoid function, and t gradually decreases during the iterative update of the weight quantization increment of the optimization unit. Floor() is the floor function.

[0149] For example, the iterative update process for V can be decomposed into the following execution steps:

[0150] 1) In floating-point mode, input image data (such as the 1024 images mentioned above) and obtain the floating-point output Y of the optimization unit. f2 (i.e., the second floating-point output mentioned above).

[0151] 2) In quantization mode, input image data (such as the 1024 images mentioned above) and obtain the quantization output Y of the optimization unit. q2 (i.e., the second quantization output mentioned above).

[0152] 3) Calculate the difference L(Y) between the optimized units. f2 Y q2 (i.e., the second quantization loss mentioned above), V is iteratively updated using the gradient descent algorithm.

[0153] 4) During the iteration process, the temperature coefficient t continuously decreases, causing the Sigmoid function to degenerate into a step function.

[0154] In the process of quantizing the weight parameters based on the target weight quantization coefficient and the target weight quantization increment, the quantization process is as follows:

[0155] Wq =Floor(W / s) + (V>0? 1:0)

[0156] Among them, W q Here, is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, Floor() is the floor function, and (V>0 ? 1 ∶ 0) means that the value is 1 when V>0 is true, and the value is 0 when V>0 is false.

[0157] In this embodiment, considering that the output features of the optimization unit are usually multi-dimensional data, and the differences in the values ​​of different dimensions may be relatively large, for example, taking 3-dimensional data as an example, the value of the first dimension may be 0 to 1, the value of the second dimension may be 10 to 100, and the value of the third dimension may be 100 to 1000, in this case, if the quantization loss is calculated directly based on the floating-point output and quantization output of the optimization unit using the mean squared error loss function, the calculation result will focus too much on the value of a certain dimension, affecting the optimization effect.

[0158] To avoid the aforementioned problems and improve optimization performance, when determining the quantization loss of the optimization unit based on its quantization output and floating-point output, for example, the aforementioned L(Y) f1 Y q1 ) or L(Y f2 Y q2 When using floating-point output features, the standard deviation of the token dimension can be calculated to balance the token difference between floating-point output features and quantized output features, and then the mean square error can be calculated.

[0159] For example, the standard deviation of the floating-point output of the optimization unit along a certain dimension can be determined first based on the floating-point output of the optimization unit. Then, the quantized output and floating-point output of the optimization unit are divided by the standard deviation to obtain the processed quantized output and processed floating-point output, and the mean square error is calculated based on the processed quantized output and processed floating-point output.

[0160] In this embodiment, the Transformer model quantized in the manner described above can be used in natural language processing tasks, such as text similarity, text classification, and machine translation; it can also be applied to speech tasks, such as speech recognition; and it can also be used for vision tasks, such as image classification, object detection, and object tracking.

[0161] The effects of the embodiments of this application will be explained below with specific application examples.

[0162] I. License Plate Recognition Scenarios

[0163] License plate recognition (LPR) is a technology that detects vehicles on monitored roads and automatically extracts and processes their license plate information (including Chinese characters, English letters, Arabic numerals, and license plate color). LPR is a crucial component of modern intelligent transportation systems and has a wide range of applications. Based on digital image processing, pattern recognition, and computer vision technologies, it analyzes vehicle images or video sequences captured by cameras to obtain a unique license plate number for each vehicle, thus completing the recognition process. Through subsequent processing, it can be used for functions such as parking fee management, traffic flow control measurement, vehicle positioning, vehicle anti-theft, automated speeding monitoring on highways, red-light violation detection, and highway toll booths. The main steps of LPR recognition are: first, locating the license plate in the image; second, segmenting the characters on the license plate; and finally, recognizing the segmented characters to assemble the license plate number.

[0164] Among them, license plate character recognition is based on neural network. The Transformer model quantized by the method provided in this application is used for photographic character recognition, which can complete the recognition task quickly and efficiently.

[0165] II. OCR Text Recognition Scenarios

[0166] OCR (Optical Character Recognition) refers to the process by which electronic devices (such as scanners or digital cameras) examine characters printed on paper and then use character recognition methods to translate the shapes into computer text; that is, scanning text materials and then analyzing and processing image files to obtain text and layout information.

[0167] Recognition speed is one of the main indicators for evaluating the performance of an OCR system. By using the method provided in this application to extract text features through quantification of the Transformer model, the recognition speed can be improved, thereby enhancing the practicality of OCR products.

[0168] III. Pedestrian Search Scenarios

[0169] Pedestrian retrieval is a technique that uses computer vision to determine whether a specific pedestrian exists in an image or video sequence; it falls under the category of image retrieval.

[0170] Given a surveillance image of a pedestrian, retrieve images of that pedestrian across different devices. The core of pedestrian retrieval lies in finding discriminative pedestrian representations. Many recent methods use deep learning models to extract visual features. By employing the Transformer model quantized using the method described in this application, fast feature extraction can be achieved, reducing time overhead.

[0171] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0172] Please see Figure 4 This is a schematic diagram of the structure of a task processing device based on model quantization provided in an embodiment of this application, as shown below. Figure 4 As shown, the model-quantization-based task processing device may include:

[0173] The first determining unit 410 is configured to, for any optimization unit in the Transformer model, update the weight quantization coefficient and activation quantization coefficient of the optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficient and target activation quantization coefficient of the optimization unit; wherein, the first quantization output is the output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input;

[0174] The second determining unit 420 is configured to, with the floating-point weight parameters of the optimization unit fixed and the target weight quantization coefficient and target activation quantization coefficient fixed, update the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target quantization coefficient and quantized according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient; and the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input.

[0175] The quantization unit 430 is used to quantize the weight parameters of the optimization unit according to the target quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, so as to obtain the target quantized weight parameters of the optimization unit.

[0176] The task processing unit 440 is configured to, when performing task processing using the Transformer model, perform forward inference calculation on the input data of any optimization unit based on the target quantization weight parameter of the optimization unit, and quantize the input / output of the optimization unit based on the target activation quantization coefficient of the optimization unit.

[0177] In some embodiments, the first determining unit 410 updates the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantization output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit, including:

[0178] Based on the first quantization output and the first floating-point output of the optimization unit, the first quantization loss of the optimization unit is determined using the first preset loss function;

[0179] Based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization coefficient and activation quantization coefficient of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.

[0180] In some embodiments, the second determining unit 420 updates the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit, including:

[0181] Based on the second quantization output and the second floating-point output of the optimization unit, the second quantization loss of the optimization unit is determined using the second preset loss function;

[0182] Based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization increment of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization increment of the optimization unit.

[0183] In some embodiments, the second determining unit 420 quantizes the weight parameters of the optimization unit according to the target weight quantization coefficient, and performs quantization adjustment according to the weight quantization increment, through the following formula:

[0184]

[0185] Where W″ is the quantized weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the weight quantization increment, t is the hyperparameter controlling the sigmoid function, and t gradually decreases during the iterative update of the weight quantization increment of the optimization unit. Floor() is the floor function.

[0186] In some embodiments, the quantization unit 430 quantizes the weight parameters of the optimization unit according to the target quantization coefficient, and determines the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, thereby obtaining the target quantized weight parameters of the optimization unit, which is achieved through the following formula:

[0187] W q =Floor(W / s) + (V>0? 1:0)

[0188] Among them, W q is the quantization weight parameter of the optimization unit, W is the floating-point weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the target weight quantization increment, Floor() is the floor function, (V>0 ? 1 ∶ 0) means that if V>0 is true, the value is 1, and if V>0 is false, the value is 0.

[0189] In some embodiments, the first determining unit 410 or the second determining unit 420 determines the quantization loss of the optimization unit based on the quantization output and floating-point output of the optimization unit using a preset loss function, including:

[0190] Based on the floating-point output of the optimization unit, determine the standard deviation of the floating-point output of the optimization unit;

[0191] The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating-point output of the optimization unit is divided by the standard deviation to obtain the processed floating-point output.

[0192] Based on the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using the mean square error loss function.

[0193] In some embodiments, the optimization unit in the Transformer model includes: Transformer Stage, Transformer Block, or a single linear layer.

[0194] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the domain-adaptive task processing method described above.

[0195] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 501 and a memory 502 storing machine-executable instructions. The processor 501 and the memory 502 can communicate via a system bus 503. Furthermore, by reading and executing the machine-executable instructions in the memory 502 corresponding to the domain-adaptive task processing logic, the processor 501 can execute the domain-adaptive task processing method described above.

[0196] The memory 502 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0197] In some embodiments, a storage medium, such as Figure 5 The memory 502 in the memory is a machine-readable storage medium that stores machine-executable instructions. When these machine-executable instructions are executed by the processor, they implement the domain-adaptive task processing method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0198] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0199] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A task processing method based on model quantization, characterized in that, Applied to computer vision, natural language processing, or speech tasks, the method includes: For any optimization unit in the Transformer model, based on the difference between the first quantized output and the first floating-point output of the optimization unit, the weight quantization coefficient and activation quantization coefficient of the optimization unit are updated with the goal of minimizing the difference, to obtain the target weight quantization coefficient and target activation quantization coefficient of the optimization unit; wherein, the first quantized output is the output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficient and the input / output of the optimization unit is quantized according to the activation quantization coefficient, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input; With the floating-point weight parameters of the optimization unit fixed and the target weight quantization coefficient and target activation quantization coefficient fixed, the weight quantization increment of the optimization unit is updated based on the difference between the second quantization output and the second floating-point output, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target weight quantization coefficient, and quantized and adjusted according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient, and the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input; The weight parameters of the optimization unit are quantized according to the target weight quantization coefficient, and the weight quantization rounding direction of the optimization unit is determined according to the target weight quantization increment, so as to obtain the target quantized weight parameters of the optimization unit. When using the Transformer model for task processing, for any optimization unit, forward inference calculation is performed on the input data of the optimization unit based on the target quantization weight parameter of the optimization unit, and the input / output of the optimization unit is quantized based on the target activation quantization coefficient of the optimization unit; the task processing includes computer vision tasks, natural language processing tasks, or speech task processing, and the input data of the Transformer model includes image data, text data, or speech data during the task processing using the Transformer model.

2. The method according to claim 1, characterized in that, The step of updating the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantized output and the first floating-point output, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit, includes: Based on the first quantization output and the first floating-point output of the optimization unit, the first quantization loss of the optimization unit is determined using the first preset loss function; Based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization coefficient and activation quantization coefficient of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit.

3. The method according to claim 1, characterized in that, The step of updating the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output, with the objective of minimizing the difference, to obtain the target weight quantization increment of the optimization unit, includes: Based on the second quantization output and the second floating-point output of the optimization unit, the second quantization loss of the optimization unit is determined using the second preset loss function; Based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization increment of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization increment of the optimization unit.

4. The method according to claim 3, characterized in that, The weight parameters of the optimization unit are quantized according to the target weight quantization coefficient, and the quantization is adjusted according to the weight quantization increment, which is achieved by the following formula: in, Here, W is the quantized and adjusted weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the weight quantization increment, and t is the hyperparameter controlling the sigmoid function. During the iterative update of the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is the floor function.

5. The method according to claim 1, characterized in that, The process involves quantizing the weight parameters of the optimization unit based on the target weight quantization coefficient, and determining the weight quantization rounding direction of the optimization unit based on the target weight quantization increment, thereby obtaining the target quantized weight parameters of the optimization unit. This is achieved through the following formula: in, Here, W is the quantization weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the target weight quantization increment, and Floor() is the floor function. This means that the value is 1 if V > 0 is true, and 0 if V > 0 is false.

6. The method according to claim 2 or 3, characterized in that, Based on the quantization and floating-point outputs of the optimization unit, the quantization loss of the optimization unit is determined using a preset loss function, including: Based on the floating-point output of the optimization unit, determine the standard deviation of the floating-point output of the optimization unit; The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating-point output of the optimization unit is divided by the standard deviation to obtain the processed floating-point output. Based on the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using the mean square error loss function.

7. The method according to any one of claims 1-5, characterized in that, The optimization units in the Transformer model include: Transformer Stage, Transformer Block, or a single linear layer.

8. A task processing device based on model quantization, characterized in that, The device, used for computer vision, natural language processing, or speech tasks, includes: The first determining unit is configured to, for any optimization unit in the Transformer model, update the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit; wherein, the first quantization output is the output of the optimization unit when the weight parameters of the optimization unit are quantized according to the weight quantization coefficients and the input / output of the optimization unit is quantized according to the activation quantization coefficients, and the first floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input; The second determining unit is configured to, with the floating-point weight parameters of the optimization unit fixed and the target weight quantization coefficient and target activation quantization coefficient fixed, update the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit; wherein, the second quantization output is the quantization output of the optimization unit when the weight parameters of the optimization unit are quantized according to the target weight quantization coefficient, and quantized and adjusted according to the weight quantization increment, and the input / output of the optimization unit is quantized according to the target activation quantization coefficient; and the second floating-point output is the floating-point output of the optimization unit when the weight parameters of the optimization unit are floating-point weight parameters and the input is a floating-point input. A quantization unit is used to quantize the weight parameters of the optimization unit according to the target weight quantization coefficient, and determine the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, so as to obtain the target quantized weight parameters of the optimization unit. The task processing unit is configured to, when performing task processing using the Transformer model, perform forward inference calculations on the input data of any optimization unit based on the target quantization weight parameter of that optimization unit, and quantize the input / output of that optimization unit based on the target activation quantization coefficient of that optimization unit; the task processing includes computer vision tasks, natural language processing tasks, or speech task processing, and the input data of the Transformer model during task processing using the Transformer model includes image data, text data, or speech data.

9. The apparatus according to claim 8, characterized in that, The first determining unit updates the weight quantization coefficients and activation quantization coefficients of the optimization unit based on the difference between the first quantized output and the first floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization coefficients and target activation quantization coefficients of the optimization unit, including: Based on the first quantization output and the first floating-point output of the optimization unit, the first quantization loss of the optimization unit is determined using the first preset loss function; Based on the first quantization loss of the optimization unit, with the goal of minimizing the first quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization coefficient and activation quantization coefficient of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization coefficient and target activation quantization coefficient of the optimization unit. The second determining unit updates the weight quantization increment of the optimization unit based on the difference between the second quantization output and the second floating-point output of the optimization unit, with the goal of minimizing the difference, to obtain the target weight quantization increment of the optimization unit, including: Based on the second quantization output and the second floating-point output of the optimization unit, the second quantization loss of the optimization unit is determined using the second preset loss function; Based on the second quantization loss of the optimization unit, with the goal of minimizing the second quantization loss, the gradient descent algorithm is used to iteratively update the weight quantization increment of the optimization unit until the preset termination condition is reached, thereby obtaining the target weight quantization increment of the optimization unit. The second determining unit quantizes the weight parameters of the optimization unit based on the target weight quantization coefficient, and adjusts the quantization based on the weight quantization increment, using the following formula: in, Here, W is the quantized weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the weight quantization increment, and t is the hyperparameter controlling the sigmoid function. During the iterative update of the weight quantization increment of the optimization unit, t gradually decreases, and Floor() is the floor function. The quantization unit quantizes the weight parameters of the optimization unit according to the target weight quantization coefficient, and determines the weight quantization rounding direction of the optimization unit according to the target weight quantization increment, thereby obtaining the target quantized weight parameters of the optimization unit, which is achieved through the following formula: in, Here, W is the quantization weight parameter of the optimization unit, s is the target weight quantization coefficient, V is the target weight quantization increment, and Floor() is the floor function. This means that the value is 1 if V > 0 is true, and 0 if V > 0 is false; Wherein, the first determining unit or the second determining unit determines the quantization loss of the optimization unit based on the quantization output and floating-point output of the optimization unit using a preset loss function, including: Based on the floating-point output of the optimization unit, determine the standard deviation of the floating-point output of the optimization unit; The quantized output of the optimization unit is divided by the standard deviation to obtain the processed quantized output; and the floating-point output of the optimization unit is divided by the standard deviation to obtain the processed floating-point output. Based on the processed quantized output and the processed floating-point output, the quantization loss of the optimization unit is determined using the mean square error loss function. The optimization units in the Transformer model include: Transformer Stage, Transformer Block, or a single linear layer.

10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-7.

11. A storage medium, characterized in that, The storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-7.

Images