A model quantization method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a model quantization method and device, electronic equipment and storage medium, comprising: determining a current target quantization layer of a current target quantization block in a target quantization model; performing inference on the target quantization model to obtain inference activation data of the current target quantization layer; quantizing weight parameters and activation data of the current target quantization layer according to a model quantization configuration file to obtain quantization data of the current target quantization layer; calculating quantization errors of the current target quantization layer according to the inference activation data and the quantization activation data; and updating quantization parameters of each quantization block according to the quantization errors of the quantization layers in each quantization block when it is determined that each quantization block of the target quantization model completes the quantization process. The technical solution of the embodiments of the present application can improve the applicability, flexibility and efficiency of the model quantization method.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer application, and particularly relate to a model quantization method and device, electronic equipment and storage medium. BACKGROUND

[0002] Model quantization is one of the most commonly used methods in model compression, and in recent years, many feasible neural network quantization strategies and schemes have emerged. However, among the numerous model quantization methods, the currently widely accepted one is INT8 (8-bit signed integer) quantization, because it can accelerate inference while not significantly reducing accuracy. However, the INT8 quantization method is relatively weak in model compression, and still does not have excellent performance in some application scenarios. In contrast, INT4 quantization can further reduce storage space and accelerate computation, but it is generally accompanied by a significant decline in model performance and even the risk of overflow.

[0003] However, existing INT4 quantization methods often require pre-quantization perception training of neural networks, which is time-consuming and complex, and requires huge computing power and time costs. At the same time, existing INT4 quantization methods are not suitable for quantization of certain types of neural network models such as LLM (Large Language Model) and the like, especially they cannot adaptively perform performance / accuracy trade-off during quantization and flexibly implement multi-precision calculation for each layer. Therefore, the applicability and flexibility of existing INT4 quantization methods are relatively low. SUMMARY

[0004] Embodiments of the present application provide a model quantization method and device, electronic equipment and storage medium, which can improve the applicability, flexibility and efficiency of the model quantization method.

[0005] According to an aspect of the present application, a model quantization method is provided, comprising:

[0006] determining a current target quantization layer of a current target quantization block in a target quantization model;

[0007] performing inference on the target quantization model to obtain inference activation data of the current target quantization layer; wherein the inference activation data includes inference input data and / or inference output data;

[0008] quantize, according to a model quantization configuration file, weight parameters and activation data of the current target quantization layer to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantized weights and quantized activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; and the quantized activation data comprises quantized input data and / or quantized output data;

[0009] calculate a quantization error of the current target quantization layer according to the inference activation data and the quantization data;

[0010] update quantization parameters of each quantization block according to the quantization error of the quantization layer in each quantization block when it is determined that the quantization process of each quantization block of the target quantization model is completed.

[0011] According to another aspect of the present application, there is provided a model quantization device, comprising:

[0012] a current target quantization layer determination module configured to determine a current target quantization layer of a current target quantization block in a target quantization model;

[0013] an inference activation data acquisition module configured to perform inference on the target quantization model to obtain inference activation data of the current target quantization layer; wherein the inference activation data comprises inference input data and / or inference output data;

[0014] a quantization data acquisition module configured to quantize, according to a model quantization configuration file, weight parameters and activation data of the current target quantization layer to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantized weights and quantized activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; and the quantized activation data comprises quantized input data and / or quantized output data;

[0015] a quantization error calculation module configured to calculate a quantization error of the current target quantization layer according to the inference activation data and the quantization data;

[0016] a quantization parameter update module configured to update quantization parameters of each quantization block according to the quantization error of the quantization layer in each quantization block when it is determined that the quantization process of each quantization block of the target quantization model is completed.

[0017] According to another aspect of the present application, there is provided an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory in communication with the at least one processor; wherein

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the model quantization method according to any one of the embodiments of the present application.

[0021] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the model quantization method according to any one of the embodiments of the present application when executed by the processor.

[0022] The embodiments of the present application determine the current target quantization layer of the current target quantization block in the target quantization model in the process of model quantization, and then perform inference on the target quantization model to obtain inference activation data of the current target quantization layer. Further, the weight parameters and activation data of the current target quantization layer are quantized according to the model quantization configuration file to obtain quantization data such as quantized weights and quantized activation data of the current target quantization layer of the current target quantization layer, so as to calculate the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data. Correspondingly, in the case that each quantization block of the target quantization model completes the quantization process, the quantization parameters of each quantization block are updated according to the quantization error of the quantization layer in each quantization block, which solves the problems of low efficiency, poor flexibility and poor applicability of the existing model quantization method, and improves the applicability, flexibility and efficiency of the model quantization method.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flowchart of a model quantization method provided by the first embodiment of the present application;

[0026] Figure 2 is a schematic diagram of a neural network model decoder structure provided by the first embodiment of the present application;

[0027] Figure 3is a flow chart of a model quantization method provided by Embodiment Two of the present application;

[0028] Figure 4 is a flow chart of a model quantization method provided by Embodiment Two of the present application;

[0029] Figure 5 is an effect diagram of compressing a 4-bit quantized weight into a uint32 element or a uint64 element provided by Embodiment Two of the present application;

[0030] Figure 6 is a flow chart of a model running method provided by Embodiment Three of the present application;

[0031] Figure 7 is a flow chart of a decompression provided by Embodiment Three of the present application;

[0032] Figure 8 is a diagram of a certain operator calculation flow in a target quantized model running process provided by Embodiment Three of the present application;

[0033] Figure 9 is a diagram of a model quantization device provided by Embodiment Three of the present application;

[0034] Figure 10 is a structural diagram of an electronic device provided by Embodiment Four of the present application. DETAILED DESCRIPTION

[0035] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the personnel in the field without creative labor should belong to the protection scope of the present application.

[0036] It should be noted that the terms "first", "second" and "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment one

[0038] Figure 1 is a flowchart of a model quantization method provided by Embodiment one of the present application. The present embodiment can be applicable to the case of quickly quantizing a single-layer network structure for a current block of a model. The method can be executed by a model quantization apparatus, which can be implemented in software and / or hardware and generally integrated in an electronic device. The electronic device can be a terminal device or a server device, as long as it can be used to process model quantization. The present application does not limit the specific type of electronic device. Correspondingly, as shown in Figure 1 the method comprises the following operations:

[0039] S110, determine a current target quantization layer of a current target quantization block in a target quantization model.

[0040] The target quantization model can be a neural network model that needs to be quantized. The current target quantization block can be a block in the target quantization model that needs to be quantized. The current target quantization layer can be a layer structure in the current target quantization block that is currently performing quantization.

[0041] In the present embodiment, the target quantization model can be a network model with an encoder and / or a decoder structure. For example, an LLM model, a Transformer model, and a visual network model with an encoder and / or a decoder structure can be used as the target quantization model. The target quantization model can include multiple blocks. A block can describe a single layer in the model, a component composed of multiple layers, or the entire model itself. Taking a decoder as an example, each block can include a decoder layer, which can include a self-attention mechanism and an MLP (Multi-layer Perceptron) module.

[0042] To improve the flexibility and efficiency of model quantization, the target quantization model can be quantized independently in a block-by-block structure. Furthermore, for each block structure that needs to be quantized, the calculation precision of each layer can be adjusted adaptively, and each layer structure in the block structure can be quantized independently without relying on other layer structures. Therefore, when quantizing the target quantization model, the blocks of the target quantization model can be traversed in sequence for quantization.

[0043] Optionally, the layer structure with the multiplication unit in the block, i.e., the layer structure of the weight, is sequentially determined as the current target quantization layer. Figure 2 is a schematic diagram of a neural network model decoder structure provided by Embodiment One of the present application. In a specific example, as shown in Figure 2 each linear (fully connected layer or dense layer) layer in the decoder structure, such as Q_linear, K_linear, V_linear, and each linear layer after the matmul (a function for tensor matrix multiplication) structure, is sequentially traversed and determined as the current target quantization layer that needs to be quantized.

[0044] Meanwhile, before quantizing the target quantization model, a model quantization configuration file applicable to the target quantization model can also be configured. The model quantization configuration file can be used to store all configuration information required in the quantization process of the target quantization model, including but not limited to the number of weight quantization bits, whether perchannel (different channels use different quantization parameters), whether symmetric quantization, and the number of input / output quantization bits, etc. Optionally, each layer structure can correspond to a model quantization configuration file, or the entire target quantization model can correspond to a model quantization configuration file, and the present application does not limit the number and scope of the model quantization configuration file.

[0045] It should be noted that the process of quantizing the target quantization model can be implemented offline, such as offline quantization based on CPU (Central Processing Unit, Central Processing Unit), and the present application does not limit this.

[0046] S120, performing inference on the target quantization model to obtain inference activation data of the current target quantization layer.

[0047] The inference activation data can include inference input data and / or inference output data. The inference input data can be input data (inputs) obtained by performing an inference operation before quantizing the current target quantization layer of the current target quantization block, and the inference output data can be output data (outputs) obtained by performing an inference operation before quantizing the current target quantization layer of the current target quantization block.

[0048] After determining the current target quantization layer, before inferring the target quantization model, a function related to statistics of input data and output data, such as a forward_hook function, can be registered first, through which inputs and outputs of the linear layer can be counted. After the function registration is completed, a subset of the validation set of the model can be selected as a calibration dataset, and the target quantization model can be input with the calibration dataset to start the inference process before quantization, so as to obtain inputs and / or outputs of the original precision, such as fp16 or fp32, of the target quantization model as inference activation data of the current target quantization layer.

[0049] In S130, weights and activation data of the current target quantization layer are quantized according to the model quantization configuration file to obtain quantization data of the current target quantization layer.

[0050] The quantization data of the current target quantization layer includes quantized weights and quantized activation data of the current target quantization layer. The activation data includes input data and / or output data of the current target quantization layer, and the quantized activation data includes quantized input data and / or quantized output data.

[0051] The quantized weights can be weights obtained by quantizing the weight parameters of the current target quantization layer. The quantized activation data can include quantized input data and / or quantized output data. The quantized input data can be data obtained by quantizing the input data of the current target quantization layer, and the quantized output data can be data obtained by quantizing the output data of the current target quantization layer.

[0052] Correspondingly, after obtaining the inference input data and / or inference output data before quantization through the inference process, specific configuration information for quantizing the weight parameters and activation data of the current target quantization layer can be determined according to the model quantization configuration file, such as bits number, perchannel, and symmetric quantization or not, and the input and / or output quantization bits number or not, and then the weight parameters and activation data of the current target quantization layer are quantized according to the specific configuration information of the weight parameters and activation data of the current target quantization layer to obtain quantization data of the current target quantization layer.

[0053] In S140, a quantization error of the current target quantization layer is calculated according to the inference activation data and the quantized activation data.

[0054] The quantization error can be used to measure the precision loss caused by the quantization operation on the target quantization model.

[0055] In the embodiments of the present application, after quantization is performed on each layer structure of the target quantization model block that needs to be quantized, the quantization error of the layer structure can be calculated according to the inference activation data and the quantized activation data of the layer structure. Optionally, when the output data and the weight parameters need to be quantized, only the quantization error of the output data can be calculated; when the input data, the output data and the weight parameters need to be quantized, the quantization errors of the input data and the output data can be calculated respectively.

[0056] Optionally, the quantization error calculated for each layer structure in the current target quantization block can be sequentially recorded in a quantization error list of the current target quantization block. Alternatively, a quantization error list can be established for each layer structure in the current target quantization block. The quantization error list can be used to record all types of quantization errors.

[0057] S150, in a case where it is determined that the quantization process of each quantization block of the target quantization model is completed, updating the quantization parameters of each quantization block according to the quantization errors of the quantization layers in each quantization block.

[0058] In the embodiments of the present application, the quantization block can be a block in the target quantization model that has completed the quantization operation. The quantization layer can be a layer in the quantization block that has completed the quantization operation. The quantization parameters are parameters related to the quantization operation, which can include, but are not limited to, the number of bits of weight quantization and the quantization scale factor, and the number of bits of quantized activation data and the quantization scale factor.

[0059] In the embodiments of the present application, when the current target quantization block completes the quantization operation, the output data of the current target quantization block after quantization can be used as the input data of the next quantization block to be quantized, the quantization error can be passed to the next layer, and the next quantization block to be quantized can be updated as the current target quantization block. The above quantization operation is repeated for the current target quantization block until all the quantization blocks to be quantized complete the quantization operation. It can be understood that after each layer structure is quantized, the quantization parameters of the layer structure can be saved.

[0060] It should be noted that through the above quantization process, low-bit quantization of the target quantization model can be achieved, such as 4-bit or 3-bit and 2-bit or lower precision low-precision information.

[0061] Correspondingly, when each layer that needs to be quantized in each block of the target quantization model is completed, all the quantization errors can be sorted, unreasonable quantization errors can be filtered out, and the layer structure corresponding to the unreasonable quantization error can be re-quantized to update the quantization parameters of the layer structure corresponding to the unreasonable quantization error, until it is determined that the quantization errors of each layer structure are kept within a reasonable range, and the final reasonable quantization parameters of each quantization layer of the target quantization model are obtained.

[0062] Therefore, the above model quantization manner takes a block as a unit, independently processes the quantization process for each layer to be quantized in each block in the model, and does not need to rely on the quantization result of other layers, which can effectively improve the quantization efficiency of the model, and is suitable for a neural network model with an encoder and / or a decoder, thereby improving the applicability, flexibility and efficiency of the model quantization method.

[0063] In the embodiment of the present application, after determining the current target quantization layer of the current target quantization block in the target quantization model in the model quantization process, the target quantization model is inferred to obtain inference activation data of the current target quantization layer. Further, the weight parameters and activation data of the current target quantization layer are quantized according to the model quantization configuration file to obtain quantization data such as quantized weights and quantized activation data of the current target quantization layer of the current target quantization layer, so as to calculate the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data. Correspondingly, in the case where the quantization process of each quantization block of the target quantization model is completed, the quantization parameters of each quantization block are updated according to the quantization error of the quantization layer in each quantization block, thereby solving the problems of low efficiency, poor flexibility and poor applicability of the existing model quantization method, and improving the applicability, flexibility and efficiency of the model quantization method.

[0064] Embodiment Two

[0065] Figure 3 is a flowchart of a model quantization method provided by the embodiment two of the present application, Figure 4 is a flowchart of a model quantization method provided by the embodiment two of the present application, and the embodiment is based on the above embodiment and is embodied. In the embodiment, various specific optional implementation manners for quantizing the weight parameters and activation data of the current target quantization layer, calculating the quantization error of the current target quantization layer, updating the quantization parameters of each quantization block, and subsequent operations of updating the quantization parameters of each quantization block are given. Correspondingly, as shown in Figure 3 and Figure 4 The method of the embodiment can include:

[0066] S210, determining a current target quantization layer of a current target quantization block in a target quantization model.

[0067] S220, inferring the target quantization model to obtain inference activation data of the current target quantization layer.

[0068] The inference activation data includes inference input data and / or inference output data.

[0069] S230, quantizing the weight parameters and activation data of the current target quantization layer according to a model quantization configuration file to obtain quantization data of the current target quantization layer.

[0070] The quantization data of the current target quantization layer includes quantization weights and quantization activation data of the current target quantization layer; the activation data includes input data and / or output data of the current target quantization layer; and the quantization activation data includes quantized input data and / or quantized output data.

[0071] In an optional embodiment of the present application, the quantization of the weight parameters and the activation data of the current target quantization layer according to the model quantization profile to obtain the quantization data of the current target quantization layer can include: determining a weight parameter quantization threshold according to the weight parameters of the current target quantization layer; calculating a first quantization scale factor of the weight parameters according to the weight parameter quantization threshold; determining a first associated quantization parameter according to the model quantization profile, and quantizing the weight parameters of the current target quantization layer according to the first associated quantization parameter and the first quantization scale factor to obtain the quantization weights; determining a second quantization scale factor of the activation data of the current target quantization layer; determining a second associated quantization parameter according to the model quantization profile, and quantizing the activation data of the current target quantization layer according to the second associated quantization parameter and the second quantization scale factor to obtain the quantization activation data.

[0072] The weight parameter quantization threshold can be a threshold for reference in quantizing the weight parameters. The first quantization scale factor can be a quantization scale factor applicable to the weight parameters. The first associated quantization parameter can be a parameter determined according to the model quantization profile and referenced in quantizing the weight parameters. The second quantization scale factor can be a quantization scale factor applicable to the activation data. The second associated quantization parameter can be a parameter determined according to the model quantization profile and referenced in quantizing the activation data. It should be noted that the types of the parameters included in the second quantization scale factor and the second associated quantization parameter need to be determined according to the types of the data included in the activation data. For example, when the activation data only includes input data, the second quantization scale factor and the second associated quantization parameter can only include relevant parameters of the input data, such as the scale of the input data and the associated quantization parameter. When the activation data only includes output data, the second quantization scale factor and the second associated quantization parameter can only include relevant parameters of the output data, such as the scale of the output data and the associated quantization parameter. When the activation data includes both input data and output data, the second quantization scale factor and the second associated quantization parameter can include relevant parameters of both the input data and the output data, such as the scale of the input data and the associated quantization parameter, and the scale of the output data and the associated quantization parameter.

[0073] Specifically, when quantizing the weight parameters and activation data of the current target quantization layer according to the model quantization configuration file, the weight parameter quantization threshold can be determined according to the weight parameters of the current target quantization layer first. For example, the absolute value maximum of the weight parameters in the current target quantization layer can be counted, and the absolute value maximum of the weight parameters in the current target quantization layer is set as the weight parameter quantization threshold in the current target quantization layer. Optionally, as shown in Figure 4 the absmax (absolute maximum quantization) of the weight parameters in the current target quantization layer can be counted by partition. Further, the scale of the weight parameters can be calculated as the first quantization scale factor according to the weight parameter quantization threshold and the associated parameters in the model quantization configuration file. For example, if the number of quantization bits of the weight parameters is 8 bits according to the associated parameters in the model quantization configuration file, the weight parameter quantization threshold can be divided by 127 to obtain the scale of the weight parameters; if the number of quantization bits of the weight parameters is 4 bits according to the associated parameters in the model quantization configuration file, the weight parameter quantization threshold can be divided by 7 to obtain the scale of the weight parameters.

[0074] After obtaining the first quantization scale factor, the first associated quantization parameters for quantizing the weight parameters can be determined according to the model quantization configuration file, such as the number of quantization bits of the weight parameters, whether perchannel, the number of groups, and whether symmetric quantization, etc. Further, the weight parameters of the current target quantization layer are quantized according to the first associated quantization parameters and the first quantization scale factor, which can specifically include but is not limited to dividing the weight parameters by the first quantization scale factor scale, and then performing round (rounding operation) and clip (limiting maximum and minimum) operations on the obtained results to obtain quantized weight parameters with a set number of bits, such as 4-bit weight. Correspondingly, after the weight parameter quantization is completed, the weight dequantization process can also be performed, such as multiplying the quantized weight by the first quantization scale factor scale to obtain the weight parameters with original precision, such as fp16 weight.

[0075] Since each linear layer has a set of input data and / or output data, each sample data also corresponds to a set of input data and / or output data, the absolute value maximum of the input data and / or output data can be counted, and the second quantization scale factor scale suitable for the activation data of the current target quantization layer can be calculated according to the absolute value maximum of the input data and / or output data. It should be noted that the second quantization scale factor scale can include the quantization scale factor scale corresponding to the input data and / or output data. That is, the second quantization scale factor scale can include the scale of one kind of data, such as the scale of input data or the scale of output data, or can include the scale of input data and the scale of output data at the same time. Similarly, after determining the second quantization scale factor scale, the second associated quantization parameter used for quantizing the activation data can be determined according to the model quantization configuration file, such as the number of quantization bits of the activation data and whether to quantize symmetrically. Similarly, the second associated quantization parameter can include the associated quantization parameter corresponding to the input data and / or output data. That is, the second associated quantization parameter can include the associated quantization parameter of one kind of data, such as the associated quantization parameter of input data or the associated quantization parameter of output data, or can include the associated quantization parameter of input data and the associated quantization parameter of output data at the same time.

[0076] Further, the activation data of the current target quantization layer is quantized according to the second associated quantization parameter and the second quantization scale factor. Specifically, it can include but is not limited to dividing the activation data by the corresponding second quantization scale factor scale, and then performing round and clip operations on the obtained results to obtain quantized input data and / or quantized output data with a set number of bits, such as 8 bits of inputs / outputs.

[0077] It should be noted that the quantization process in the embodiment of the present application can be set according to actual needs. For example, only the output data can be quantized, and the input data and the weight parameter are not quantized. At this time, the output data obtained by inference can be directly quantized to obtain quantized output data.

[0078] Optionally, the input data and / or the weight can also be quantized, and the output data is not quantized. At this time, the input data and / or the weight can be directly quantized to obtain quantized input data and / or quantized weight.

[0079] Optionally, if quantization is needed for the output data, while quantization is also needed for the input data and / or the weight, the input data and / or the weight can be quantized first to obtain quantized input data and / or quantized weight, and then the input data and / or the weight can be dequantized to obtain dequantized input data and / or dequantized weight, and then the output data can be quantized according to the dequantized input data and / or the dequantized weight. Specifically, if only the input data is quantized, the dequantized input data and the result of the quantized weight parameter (not quantized) can be multiplied, and the result of the multiplication can be quantized to obtain quantized output data. If only the weight parameter is quantized, the input data (not quantized) and the result of the dequantized weight can be multiplied, and the result of the multiplication can be quantized to obtain quantized output data. If both the weight parameter and the input data are quantized, the result of the dequantized input data and the dequantized weight can be multiplied, and the result of the multiplication can be quantized to obtain quantized output data.

[0080] In a specific example, in the process of quantization, when the activation data only includes the input data, the input data can be divided by the scale of the input data, and the result can be subjected to round and clip operations to obtain quantized input data of a set number of bits. When the activation data only includes the output data, the output data can be divided by the scale of the output data, and the result can be subjected to round and clip operations to obtain quantized output data of a set number of bits. When the activation data includes both the input data and the output data, the input data can be divided by the scale of the input data, and the result can be subjected to round and clip operations to obtain quantized input data of a set number of bits, and the output data can be divided by the scale of the output data, and the result can be subjected to round and clip operations to obtain quantized output data of a set number of bits.

[0081] Correspondingly, after the quantization of the activation data is completed, the dequantization process of the activation data can also be performed. For example, the quantized activation data can be multiplied by the second quantization scale factor scale to obtain dequantized activation data, i.e., input data and / or output data of original precision, such as fp16 inputs / outputs.

[0082] It should be noted that the quantization bit number of the activation data can be different from that of the weight parameter, and the quantization bit number of the input data and the output data included in the activation data can also be different. For example, the quantization bit number of the weight parameter can be 4, and the quantization bit number of the activation data can be 8 or 16, and the embodiments of the present application do not limit this.

[0083] Correspondingly, if the activation data includes the input data and the output data of the current target quantization layer, and the quantized activation data includes the quantized input data and the quantized output data, the subsequent process of the model quantization method can include the following operations:

[0084] In S240, the quantized weight of the current target quantization layer is dequantized to obtain a dequantized weight, and the quantized input data of the current target quantization layer is dequantized to obtain dequantized input data.

[0085] In S250, a product value of the dequantized input data and the dequantized weight is calculated to obtain an intermediate output data, and the intermediate output data is quantized and dequantized to obtain dequantized output data.

[0086] The dequantized weight can be data obtained by dequantizing the quantized weight of the current target quantization layer. The dequantized input data can be data obtained by dequantizing the quantized input data of the current target quantization layer. The dequantized output data can be data obtained by quantizing and dequantizing the intermediate output data. It should be noted that when only the output data is quantized, the dequantized output data can be data obtained by quantizing the original output data to obtain a quantized result and then dequantizing the quantized result. When the input data and / or the weight data are quantized, the dequantized output data can be data obtained by quantizing the intermediate output data to obtain a quantized result and then dequantizing the quantized result.

[0087] In the embodiment of the present application, when calculating the quantization error of the current target quantization layer, if the activation data includes the input data and the output data of the current target quantization layer, and the quantized activation data includes the quantized input data and the quantized output data, it means that the input data and the output data are quantized at the same time. At this time, the quantized weight of the current target quantization layer can be dequantized, that is, the quantized weight is multiplied by a first quantization scale factor to obtain a dequantized weight. At the same time, the quantized input data of the current target quantization layer can be dequantized to obtain dequantized input data, that is, the quantized input data is multiplied by a second quantization scale factor corresponding to the input data to obtain the dequantized input data. In addition, the product value of the dequantized input data and the dequantized weight can be calculated to obtain the intermediate output data, and the intermediate output data is quantized and dequantized, that is, the intermediate output data is first divided by the corresponding second quantization scale factor, and the obtained result is subjected to round and clip operations to obtain the quantized output data with a certain number of bits, and then the quantized output data is multiplied by the second quantization scale factor corresponding to the output data to obtain the dequantized output data.

[0088] S260, calculate the difference between the inference input data and the dequantization input data to obtain the quantization error of the input data, and calculate the difference between the inference output data and the dequantization output data to obtain the quantization error of the output data.

[0089] In the embodiment of the present application, after the quantized weight and the quantized activation data are dequantized, the quantization error of the current target quantization layer can be calculated. Specifically, the quantization error under different quantization configurations can be counted. For example, the different quantization configurations can be whether the inputs are quantized in the case of quantizing the weight, whether the outputs are quantized, and the like. For example, if the inputs and the outputs are both quantized in the case of quantizing the weight, the quantization error of the inputs and the outputs can be calculated respectively. Optionally, the way of calculating the quantization error can be the difference between the dequantized data and the data before quantization, such as calculating the difference between the inference input data and the dequantized input data to obtain the quantization error of the input data, and calculating the difference between the inference output data and the dequantized output data to obtain the quantization error of the output data. Optionally, the difference can be represented by an index type such as SNR (Signal-to-noise ratio) or MSE (Mean Square Error). For example, the SNR or MSE calculated from the dequantized activation data and the inference activation data can be used as the quantization error of the current target quantization layer.

[0090] Optionally, after the quantization error of the current target quantization layer is calculated, the quantization error of the current target quantization layer can be added to the quantization error list for storage. At the same time, the quantization parameters of the current target quantization layer can be saved, such as weight_scale (the first quantization scale factor corresponding to the quantized weight) / weight_bits (the number of bits of the quantized weight), input_scale (the second quantization scale factor corresponding to the quantized input data) / input_bits (the number of bits of the quantized input data), output_scale (the second quantization scale factor corresponding to the quantized output data) / output_bits (the number of bits of the quantized output data), and the like. Correspondingly, after the quantization parameters of the current target quantization layer are saved successfully, the occupied GPU memory of the current target quantization layer can be released.

[0091] It should be noted that in order to further improve the quantization efficiency, each quantization scale factor can be uniformly collected into one quantization scale factor for storage. For example, the first quantization scale factor corresponding to the quantization weight, the second quantization scale factor corresponding to the quantization input data, and the second quantization scale factor corresponding to the quantization output data can be integrated into one quantization scale factor for storage. When the integrated quantization scale factor is used for quantization operation and dequantization operation, the quantization scale factor corresponding to each data can be determined through the integrated quantization scale factor to perform data calculation process. For example, the quantization scale factor corresponding to the weight parameter can be determined through the integrated quantization scale factor to perform the quantization and dequantization process of the weight parameter.

[0092] At this point, the quantization process of the current target quantization layer is completed. If there are still layer structures that need to be quantized in the current target quantization block, the next layer structure to be quantized can be updated as the current target quantization layer, and the operation of performing inference on the target quantization model is returned until it is determined that all layer structures that need to be quantized in the current target quantization block are completed.

[0093] After storing the corresponding quantization parameters of all the layer structures in the current target quantization block that need quantization processing, in order to truly realize the quantization process of the current target quantization block by using the quantization parameters, the related hook functions such as the forward_pre_hook (another kind of hook function) and forward_hook functions can be registered to quantize the inputs and outputs of the linear layer in the current target quantization block. Further, according to the stored quantization parameters such as the weight_scale / weight_bits of the quantization weight corresponding to each layer structure in the current target quantization block, the input_scale / input_bits of the quantized input data, and the output_scale / output_bits of the quantized output data, etc., the weight / inputs / outputs in the current target quantization block can be added with Quant (quantization) / Dequant (dequantization), and the target quantization model is inferred for the second time according to the quantization parameters to obtain the output data outputs of the current target quantization block after quantization. After obtaining the output data outputs of the current target quantization block after quantization, the occupied video memory of the current target quantization block can be released, and the output data outputs of the current target quantization block after quantization are assigned to the inputs of the next block that needs quantization processing, the quantization error is transmitted to the next layer, and the next block that needs quantization processing is updated as the current target quantization block, and the operation of determining the current target quantization layer of the current target quantization block in the target quantization model is returned to execute. The above quantization operation is repeated for the updated current target quantization block until all the block structures in the target quantization model complete the quantization process.

[0094] S270, determine whether each quantization block of the target quantization model completes the quantization process, if yes, execute S280, otherwise, return to execute S210.

[0095] S280, update the quantization parameters of each quantization block according to the quantization error of the quantization layer in each quantization block.

[0096] In an optional embodiment of the present application, the updating of the quantization parameters of each quantization block according to the quantization error of the quantization layer in each quantization block can comprise: determining a quantization error threshold; determining the quantization layer whose quantization error is greater than the quantization error threshold as a to-be-updated quantization layer; modifying the quantization configuration parameter of the to-be-updated quantization layer in the model quantization configuration file, and determining the to-be-updated quantization layer as the current target quantization layer, and returning to perform the operation of quantizing the weight parameter and the activation data of the current target quantization layer according to the model quantization configuration file until the quantization error of the to-be-updated quantization layer is determined to be less than or equal to the quantization error threshold; saving the quantization parameters of the quantization layer in each quantization block; wherein the quantization parameters comprise the quantization scale factor of the weight parameter, the bits of the quantized weight parameter, the quantization scale factor of the activation data, and the bits of the quantized activation data.

[0097] The quantization error threshold can be used to screen unreasonable quantization errors, and the specific value of the quantization error threshold can be set according to actual needs, and the present application does not limit the specific value of the quantization error threshold. The to-be-updated quantization layer can be a layer structure that needs to be updated with quantization parameters.

[0098] In the embodiment of the present application, after all the blocks in the target quantization model complete the above quantization process, unreasonable quantization parameters can be screened out for re-quantization. Specifically, a quantization error threshold can be first determined, and all quantization errors in the quantization error list are compared with the quantization error threshold to screen out quantization errors whose values exceed the quantization error threshold, and the quantization layer whose quantization error is greater than the quantization error threshold is determined as a to-be-updated quantization layer. Further, the quantization configuration parameter of the to-be-updated quantization layer in the model quantization configuration file is modified, for example, the number of quantization blocks is increased, the number of quantization bits is increased, the accuracy of quantized activation data is improved, etc. After the modification of the quantization configuration parameter of the to-be-updated quantization layer is completed, the to-be-updated quantization layer can be determined as the current target quantization layer, and the operation of quantizing the weight parameter and the activation data of the current target quantization layer according to the model quantization configuration file is returned to be performed until the quantization error of the to-be-updated quantization layer is determined to be less than or equal to the quantization error threshold. When the quantization error of the to-be-updated quantization layer is determined to be less than or equal to the quantization error threshold, it indicates that the to-be-updated quantization layer has completed the reasonable quantization operation, and at this time, the quantization parameters of the quantization layer in each quantization block, i.e., weight_scale / weight_bits, input_scale / input_bits, and output_scale / output_bits, can be saved as the final quantization parameters.

[0099] In a specific example, as shown in FIG. 2, the model quantization process can specifically comprise the following flow: Figure 4 ​

[0100] Step one, traverse Block quantization in turn:

[0101] Step two, quantize the weight of the layer with a multiplication unit in the single Block in turn, build a single-layer quantization layer, and configure its quantization config (i.e., model quantization configuration file). The quantization config can include but is not limited to the number of weight quantization bits, whether perchannel, whether symmetric quantization, and the number of input / output quantization bits, etc.

[0102] Step three, register forward_hook to count the inputs and outputs of the linear layer.

[0103] Step four, first inference: perform model inference process through calibration dataset (calibration set) to obtain the inputs and outputs of the linear layer in actual original precision (such as fp16 or fp32).

[0104] Step five, count the maximum absolute value of weight as the quantization threshold (threshold), for example, the absmax of weight can be counted according to the partition, and the scale of quantized weight is calculated according to the quantization threshold and the quantization config.

[0105] Step six, Weight Quant (weight quantization): according to the quantization config, the weight is quantized, specifically, the weight weight is divided by the scale of quantized weight, and after round and clip operations, the quantized 4-bit weight is obtained.

[0106] Step seven, Weight Dequant (weight dequantization): the weight weight is multiplied by the scale of quantized weight to obtain the fp16 weight, and the weight dequantized weight can be used for subsequent calculation of quantization error.

[0107] Step eight, since each layer of linear layer has a set of inputs / outputs, and each sample data also corresponds to a set of inputs / outputs, the absolute maximum value of inputs / outputs can be counted.

[0108] Step nine, Activation Quant (Activation data quantization): According to the quantization config file, divide the activation data Activation by the scale of quantized activation data, and perform operations such as round and clip to obtain quantized inputs / outputs. The quantized inputs / outputs are generally 4 bits, 8 bits or fp16, which can be different from the weight.

[0109] Step ten, Activation DeQuant (Activation data dequantization): Multiply the quantized activation data inputs / outputs by the corresponding scale to obtain fp16 inputs / outputs. The scales corresponding to inputs and outputs can be the same or different.

[0110] Step eleven, statistics of quantization error error of quantized weights / inputs / outputs under different quantization configurations (i.e. whether to quantize inputs and whether to quantize outputs under the condition of quantizing weights), including but not limited to SNR and MSE, are added to the quantization error list error_list of the layer.

[0111] Step twelve, save the quantization parameters of a single layer layer: weight_scale / weight_bits, input_scale / input_bits, output_scale / output_bits.

[0112] Step thirteen, release the memory occupied by the layer.

[0113] Step fourteen, repeat steps two to thirteen to complete the quantization of all layers in a single block.

[0114] Step fifteen, select quantized inputs / outputs according to the quantization config.

[0115] Step sixteen, register forward_pre_hook and forward_hook for quantizing the inputs and outputs of the linear layer.

[0116] Step seventeen, second inference: according to the quantization configuration, add Quant / Dequant (i.e. quantization and dequantization) to weight / inputs / outputs according to the weight_scale / weight_bits, input_scale / input_bits, output_scale / output_bits calculated in the first inference, and then perform inference according to the quantization parameters to obtain the outputs of the block.

[0117] Step eighteen, release the memory occupied by the block.

[0118] Step nineteen, assign the outputs of the block to the inputs of the next block, pass the quantization error to the next layer, and repeat steps one to eighteen (calculate the quantization parameters of the next block, quantize the weight / input / output of the block).

[0119] Step twenty, after completing the quantization of all blocks, sort the layers according to the statistical error_list and select the layers whose error exceeds the threshold error.

[0120] Step twenty-one, modify the layer quantization config, including but not limited to increasing the number of quantization blocks and the number of quantization bits when quantizing weights, using higher precision for inputs / outputs, etc. Re-quantize the layer and calculate the error.

[0121] Step twenty-two, determine whether the error of the selected layer is greater than the threshold error, if yes, repeat step twenty-one; otherwise, execute step twenty-two.

[0122] Step twenty-three, update and save the final reasonable and available quantization parameters of the selected layer: weight_scale / weight_bits, input_scale / input_bits, and output_scale / output_bits.

[0123] The above technical solution employs a block-by-block quantization model, reducing memory usage during the quantization process. Special functions (such as mean square error) are introduced to compensate for quantization accuracy when calculating quantization errors. The quantization data types for each layer's weights and inputs / outputs can be adaptively adjusted based on the error. Furthermore, based on the quantization error, a hook registration mechanism can adaptively adjust the quantization data type of input activation data for different dot values ​​(dot functions are mainly used for vector dot products and matrix multiplications). This converts dot calculations from floating-point multiplication to int4*int4, int4*int8, int4*fp16, or int8*int8, flexibly configuring the optimal inference scheme for the accelerated model based on the hardware architecture. Simultaneously, the quantization method allows configuration of the precision of the output activation data, quantizing the output to reduce output I / O (Input / Output). During quantization, activation data can be calculated using floating-point numbers, thus preserving the model's inference accuracy.

[0124] The aforementioned model quantization method does not require quantization training of the model and can be implemented in hardware or firmware. By adaptively adjusting the computational precision of each layer, it can flexibly adapt to model structures such as Decoder and Encoder, and is particularly suitable for LLM models such as GPT (Generative Pre-Trained Transformer), OPT (Open Pre-trained Transformer Language Models), and LLMA (Large Language Model Meta AI). Compared with model quantization methods based on quantization training, the quantization efficiency is greatly improved. Through post-training quantization, the matrix multiplication precision can be converted from the original fp32 / fp16 to int4*int8, int4*fp16, or int8*int8, etc., reducing the storage space occupied by the weights to 1 / 8 of the original, reducing computational memory, and configuring matching int42D and 1D computing power can significantly improve the model's inference performance. Verification was conducted using the llama and OPT models in the GPT class. After quantization using the above model quantization method, the W4A16 operator showed a decrease in accuracy of less than 1%, a reduction in model parameters to 1 / 4 of the fp16 model, and a reduction in latency of more than 3x compared to fp16. When extended to the mixed precision of the W4A16 / W4A8 operators, the performance of the quantized model can be further improved.

[0125] In an optional embodiment of the present application, after the quantization error of each quantization layer is updated according to the quantization parameter of each quantization block, the method further comprises: determining a target data type supported by a hardware structure of a target device running the target quantization model; and compressing the quantization weight of the target quantization model according to the target data type supported by the hardware structure of the target device.

[0126] The target device can be a device running the target quantization model after quantization processing, and can be any type of terminal device, such as a smart phone, a smart detection system terminal, a tablet computer, a personal computer, or the like. The target device can also be a server device, as long as there is a demand for running the target quantization model after quantization processing. The specific device type of the target device is not limited in the embodiments of the present application. The target data type can be a data format type supported by the hardware structure of the target device.

[0127] In order to further save the on-chip memory occupancy of the running model device and reduce the loading time of the model, the quantization weight of the target quantization model can be compressed before the target quantization model after quantization is loaded into the target device for running. It can be understood that the hardware structure of some types of electronic devices can support limited data format types. For example, the hardware structure of some devices supports int4 data format, while the hardware structure of some devices does not support int4 data format. Therefore, when the quantization weight of the target quantization model is compressed, the target data type supported by the hardware structure of the target device running the target quantization model can be determined, and the quantization weight of the target quantization model can be compressed according to the target data type supported by the hardware structure of the target device, so as to adapt to the efficient decompression operation when different target devices run.

[0128] In an optional embodiment of the present application, the compression processing of the quantization weights of the target quantization model according to the target data type supported by the hardware structure of the target device can comprise: in the case that it is determined that the target data type supported by the hardware structure of the target device comprises int4, determining the first quantization weight in the current group of quantization weights to be compressed and the bit number of each quantization weight in the current group of quantization weights to be compressed; in the case that it is determined that the first quantization weight is positive, performing left shift processing on the current quantization weight to be compressed according to the bit number of each quantization weight, and then performing bit-by-bit or processing on the previous compressed quantization weight result of the current quantization weight to be compressed to obtain the current compressed quantization weight; in the case that it is determined that the first quantization weight is negative, performing complement processing on the first quantization weight according to the bit number of each quantization weight, and then performing left shift processing on the current quantization weight to be compressed and low-bit padding processing to obtain the padded quantization weight, and then performing bit-by-bit and processing on the padded quantization weight and the previous compressed quantization weight result of the current quantization weight to be compressed to obtain the current compressed quantization weight.

[0129] In the formula, the current group of quantization weights to be compressed can be a group of quantization weights that need to be compressed. For example, the current group of quantization weights to be compressed can comprise 8 quantization weights to be compressed. The number of quantization weights included in the current group of quantization weights to be compressed is not limited in the embodiments of the present application.

[0130] Optionally, if the hardware structure of the target device supports the int4 data format, when the quantization weight is quantized to 4 bits or less, the quantization weight can be compressed (packed) into uint32 or uint64 for storage and loading. Figure 5 is a schematic diagram of the effect of compressing a 4-bit quantization weight into a uint32 element or a uint64 element according to the second embodiment of the present application. For example, as shown in Figure 5As shown, taking 4-bit weight as an example, the process of compressing 8 int4 weights into 1 uint32 data specifically includes the following operations: all quantized weights are divided into groups of 8, and compression processing is performed in groups. Correspondingly, if the quantized weight of the first int4 in the current quantized weight group to be compressed is positive, the quantized weight of the Nth int4 can be sequentially left shifted by 4*(N-1) bits, and then the weight group result after compression processing of the (N-1)th quantized weight in the current quantized weight group to be compressed is bitwise taken | to obtain the packed uint32 weight data. Correspondingly, if the quantized weight of the first int4 in the current quantized weight group to be compressed is negative, the quantized int4 weight is stored as int32 in complement form, then the quantized weight of the Nth int4 can be sequentially left shifted by 4*(N-1) bits, and the low bits are supplemented with 1, and then the weight group result after compression processing of the (N-1)th quantized weight in the current quantized weight group to be compressed is bitwise taken & to obtain the packed uint32 weight data.

[0131] Optionally, if the hardware structure of the target device does not support the data format of int4, the quantized weight can be converted into int8 for storage. Specifically, by performing an interleave operation on two groups of 8 int4 quantized weights (taking every other one), the 8 uint8 format weight data is arranged into uint64 weight data for efficient decompression processing and parallel computing in the runtime stage.

[0132] In an optional embodiment of the present application, after the quantized weight of the target quantized model is compressed according to the target data type matched by the target device, the method can further include: loading the quantized data of the current quantized layer of the target quantized model and the target quantized reference parameter during the process of running the target quantized model on the target device. According to the quantized results of the quantized input data and the quantized weight in the quantized data, and in combination with the target quantized reference parameter, data calculation is performed in the current quantized layer.

[0133] The target quantized model is obtained by the model quantization method described in any of the above embodiments. The current quantized layer can be a layer structure that is currently performing data calculation in the process of running the target quantized model on the target device. The target quantized reference parameter can be a quantization parameter used for decompression processing of the quantized data of the current quantized layer, which can be a quantization scale factor, for example. Optionally, the target quantized reference parameter can be a first quantization scale factor corresponding to the quantized weight, and a second quantization scale factor of each layer activation data.

[0134] In the embodiment of the present application, when the target quantization model is quantized by the model quantization method of any of the above embodiments and the quantized weight obtained by quantization is compressed, it can be loaded into the target device to run the application. In the process of running the target quantization model in the target device, the quantization data of the current quantization layer of the target quantization model and the target quantization reference parameters can be loaded, such as the compressed quantized weight, the scale parameter corresponding to the quantized weight, and the scale parameter corresponding to the activation data, and the calculation of each layer is performed in turn. Optionally, the hardware structure of the target device can support any type of data type, such as int4 data format, etc.

[0135] In an optional embodiment of the present application, the data calculation in the current quantization layer according to the quantization results of the quantized input data and the quantized weight in the quantization data, in combination with the target quantization reference parameters can include: in the case where the quantization results of the quantized input data and the quantized weight are both int4, performing four-bit matrix multiplication on the quantized input data and the quantized weight, and performing product operation on the matrix multiplication result and the target quantization reference parameters to obtain the current quantization layer output data; in the case where the quantization result of the quantized input data is int8 and the quantization result of the quantized weight is int4, decompressing the quantized weight into int8 data type, then performing eight-bit integer multiplication on the quantized input data and the decompressed quantized weight, and performing product operation on the multiplication result and the target quantization reference parameters to obtain the current quantization layer output data; in the case where the quantization result of the quantized input data is fp16 and the quantization result of the quantized weight is int4, decompressing the quantized weight into int8 data type, performing product operation on the decompressed quantized weight and the target quantization reference parameters, then performing matrix multiplication on the quantized input data and the quantized weight after product operation, and taking the matrix multiplication result as the current quantization layer output data; taking the current quantization layer output data or the quantization result of the current quantization layer output data as the quantized input data of the next quantization layer, and outputting it to the next quantization layer.

[0136] Among them, the current quantization layer output data can be the data output by the current quantization layer.

[0137] Figure 6 is a flowchart of a model running method provided in Embodiment Three of the present application, Figure 7 is a flowchart of a decompression method provided in Embodiment Three of the present application, Figure 8 is a schematic diagram of an operator calculation process in the running process of a target quantization model provided in Embodiment Three of the present application. In a specific example, as shown in Figure 6 , Figure 7 and Figure 8As shown, when the target device runs the target quantization model and performs data calculation in the current quantization layer of the target quantization model, the bit number of the quantization result of the quantized input data input and the quantization result of the quantized weight weight can be determined. If it is determined that the quantization result of the quantized input data input and the quantization result of the quantized weight weight are both int4, and the target device supports int4 data format, then the input and the weight can be multiplied by four-bit matrix through int4 matrix multiplication instruction such as nvidia mma (NVIDIA multiplication operation instruction) instruction, and the matrix multiplication result is multiplied by the scale parameter to obtain the output data of the current quantization layer.

[0138] If it is determined that the quantization result of the quantized input data input is int8 and the quantization result of the quantized weight weight is int4, then the weight can be first decompressed twice to decompress the weight into int8 data type. For example, when the int4 weight is decompressed, the low 4 bits of the int4 weight can be logically left shifted by 4 bits and then arithmetically right shifted by 4 bits, and the high 4 bits of the int4 weight can be arithmetically right shifted by 4 bits. Further, the quantized input data input and the decompressed quantized weight weight are multiplied by eight-bit matrix through int8 matrix multiplication instruction, and the matrix multiplication result is multiplied by the scale parameter to obtain the output data of the current quantization layer.

[0139] If it is determined that the quantization result of the quantized input data input is fp16 and the quantization result of the quantized weight weight is int4, then the weight can be first decompressed through shifting and bit operation to decompress the weight into int8 data type. Further, the decompressed quantized weight weight is multiplied by the scale parameter to obtain the fp16 weight, and the fp16 quantized input data input and the fp16 weight are multiplied by matrix to obtain the matrix multiplication result as the output data of the current quantization layer.

[0140] In the above examples, the scale parameter can be a scale parameter integrated from the scale parameter of the quantized weight, the scale parameter of the quantized input data, and the scale parameter of the quantized output data. Meanwhile, each instruction can be implemented through hardware instruction.

[0141] Correspondingly, the current quantization layer output data can be output to the next layer input, and the next quantization layer can be updated to the current quantization layer. In the current quantization layer, if the quantization result of the quantization input data and the quantization weight is determined to be int4, the operation of multiplying the quantization input data and the quantization weight by a four-bit matrix can be performed until all layers have completed the data calculation process.

[0142] It should be noted that if the model quantization configuration file contains quantization-related configuration information for the current quantization layer output data, then after obtaining the current quantization layer output data, the current quantization layer output data can be quantized according to the model quantization configuration file, and the quantization result of the current quantization layer output data can be used as the quantization input data for the next quantization layer and output to the next quantization layer.

[0143] The above technical solution compresses the quantization weights and decompresses them using hardware instructions during the execution of the target quantization model. This achieves a co-design of software and hardware, enabling the target quantization model algorithm to be executed efficiently on hardware, thereby achieving more efficient hardware performance.

[0144] The model quantization method provided in this invention does not require quantization training of the model. It can be implemented in hardware or firmware. It quantizes layer by layer according to the model block. The quantization process reduces the memory usage and is flexibly adaptable to models such as GPT, OPT and llama. It can significantly improve the efficiency, applicability and flexibility of model quantization.

[0145] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0146] Example 3

[0147] Figure 9 This is a schematic diagram of a model quantization device provided in Embodiment 3 of the present invention, as shown below. Figure 9 As shown, the device includes: a current target quantization layer determination module 310, an inference activation data acquisition module 320, a quantization data acquisition module 330, a quantization error calculation module 340, and a quantization parameter update module 350, wherein:

[0148] The current target quantization layer determination module 310 is used to determine the current target quantization layer of the current target quantization block in the target quantization model;

[0149] The inference activation data acquisition module 320 is used to perform inference on the target quantization model to obtain the inference activation data of the current target quantization layer; wherein, the inference activation data includes inference input data and / or inference output data;

[0150] The quantization data obtaining module 330 is configured to quantize the weight parameters and the activation data of the current target quantization layer according to the model quantization configuration file to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantized weights and quantized activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; and the quantized activation data comprises quantized input data and / or quantized output data.

[0151] The quantization error calculating module 340 is configured to calculate the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data.

[0152] The quantization parameter updating module 350 is configured to update the quantization parameters of each quantization block according to the quantization errors of the quantization layers in each quantization block when it is determined that the quantization process of each quantization block of the target quantization model is completed.

[0153] In the embodiments of the present application, after the current target quantization layer of the current target quantization block in the target quantization model is determined in the model quantization process, the target quantization model is inferred to obtain the inference activation data of the current target quantization layer. Further, the weight parameters and the activation data of the current target quantization layer are quantized according to the model quantization configuration file to obtain the quantization data such as the quantized weights and the quantized activation data of the current target quantization layer of the current target quantization layer, so as to calculate the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data. Accordingly, when it is determined that the quantization process of each quantization block of the target quantization model is completed, the quantization parameters of each quantization block are updated according to the quantization errors of the quantization layers in each quantization block, thereby solving the problems of low efficiency, poor flexibility and poor applicability existing in the prior model quantization method, and improving the applicability, flexibility and efficiency of the model quantization method.

[0154] Optionally, the quantization data obtaining module 330 is specifically configured to determine a weight parameter quantization threshold according to the weight parameters of the current target quantization layer.

[0155] The first quantization scale factor of the weight parameters is calculated according to the weight parameter quantization threshold.

[0156] The first associated quantization parameter is determined according to the model quantization configuration file, and the weight parameters of the current target quantization layer are quantized according to the first associated quantization parameter and the first quantization scale factor to obtain the quantized weights.

[0157] The second quantization scale factor of the activation data of the current target quantization layer is determined.

[0158] determine a second associated quantization parameter according to the model quantization configuration file, and quantize the activation data of the current target quantization layer according to the second associated quantization parameter and the second quantization scale factor to obtain the quantized activation data.

[0159] Optionally, if the activation data includes input data and output data of the current target quantization layer, and the quantized activation data includes quantized input data and quantized output data, the quantization error calculation module 340 is specifically configured to: perform dequantization processing on the quantized weight of the current target quantization layer to obtain a dequantized weight;

[0160] perform dequantization processing on the quantized input data of the current target quantization layer to obtain dequantized input data;

[0161] calculate a product value of the dequantized input data and the dequantized weight to obtain an intermediate value of output data;

[0162] perform quantization processing and dequantization processing on the intermediate value of output data to obtain dequantized output data;

[0163] calculate a difference value between the inference input data and the dequantized input data to obtain a quantization error of input data;

[0164] calculate a difference value between the inference output data and the dequantized output data to obtain a quantization error of output data.

[0165] Optionally, the quantization parameter updating module 350 is specifically configured to: determine a quantization error threshold;

[0166] determine a quantization layer whose quantization error in each quantization block is greater than the quantization error threshold as a to-be-updated quantization layer;

[0167] modify the quantization configuration parameter of the to-be-updated quantization layer in the model quantization configuration file, and determine the to-be-updated quantization layer as the current target quantization layer, and return to perform the operation of quantizing the weight parameter and the activation data of the current target quantization layer according to the model quantization configuration file until the quantization error of the to-be-updated quantization layer is less than or equal to the quantization error threshold;

[0168] save the quantization parameters of the quantization layers in each quantization block; wherein the quantization parameters include a quantization scale factor of a weight parameter and a bit position of the quantized weight parameter, and a quantization scale factor of activation data and a bit position of the quantized activation data.

[0169] Optionally, the model quantization device further includes a quantized weight compression processing module configured to: determine a target data type supported by a hardware structure of a target device running the target quantization model;

[0170] According to the target data type supported by the hardware structure of the target device, the quantization weights of the target quantization model are compressed.

[0171] Optionally, the quantization weight compression processing module is specifically configured to: in a case where it is determined that the target data type supported by the hardware structure of the target device includes int4, determining a first quantization weight in the current quantization weight group to be compressed and bit numbers of each quantization weight in the current quantization weight group to be compressed;

[0172] In a case where it is determined that the first quantization weight is a positive number, performing left shift processing on the current quantization weight to be compressed according to the bit numbers of each quantization weight, and then performing bit-by-bit or processing on the current quantization weight to be compressed and a previous compressed quantization weight result of the current quantization weight to be compressed to obtain a current compressed quantization weight;

[0173] In a case where it is determined that the first quantization weight is a negative number, performing complement processing on the first quantization weight according to the bit numbers of each quantization weight, and then performing left shift processing on the current quantization weight to be compressed and low-bit padding processing to obtain a padded quantization weight, and performing bit-by-bit and processing on the padded quantization weight and a previous compressed quantization weight result of the current quantization weight to be compressed to obtain a current compressed quantization weight.

[0174] Optionally, the model quantization device further includes a model data calculation module configured to: in a process in which the target device runs the target quantization model, loading quantization data and a target quantization reference parameter of a current quantization layer of the target quantization model;

[0175] According to the quantization results of the quantization input data and the quantization weight in the quantization data, and in combination with the target quantization reference parameter, data calculation is performed in the current quantization layer.

[0176] Optionally, the model data calculation module is specifically configured to: in a case where it is determined that the quantization input data and the quantization result of the quantization weight are both int4, performing four-bit matrix multiplication on the quantization input data and the quantization weight, and performing product operation on a matrix multiplication result and the target quantization reference parameter to obtain current quantization layer output data;

[0177] In a case where it is determined that the quantization result of the quantization input data is int8 and the quantization result of the quantization weight is int4, decompressing the quantization weight into data of the int8 type, performing eight-bit integer multiplication on the quantization input data and the decompressed quantization weight, and performing product operation on a multiplication result and the target quantization reference parameter to obtain current quantization layer output data;

[0178] In a case where it is determined that the quantization result of the quantized input data is fp16 and the quantization result of the quantized weight is int4, the quantized weight is decompressed into a data type of int8, a product operation is performed on the decompressed quantized weight and the target quantization reference parameter, a matrix multiplication is performed on the quantized input data and the quantized weight after the product operation, and the matrix multiplication result is taken as current quantization layer output data;

[0179] The current quantization layer output data or the quantization result of the current quantization layer output data is taken as quantization input data of a next quantization layer and output to the next quantization layer.

[0180] The model quantization apparatus can perform the model quantization method provided by any embodiment of the present application, has the function modules and beneficial effects corresponding to the execution method. Technical details not described in detail in the present embodiment can refer to the model quantization method provided by any embodiment of the present application.

[0181] Since the model quantization apparatus described above is an apparatus that can perform the model quantization method in the embodiments of the present application, based on the model quantization method described in the embodiments of the present application, those skilled in the art can understand the specific implementation of the model quantization apparatus of the present embodiment and its various forms, so the model quantization apparatus how to implement the model quantization method in the embodiments of the present application will not be described in detail here. As long as the apparatus used to implement the model quantization method in the embodiments of the present application is implemented by those skilled in the art, it belongs to the scope of the present application.

[0182] Embodiment Four

[0183] Figure 10 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document.

[0184] As Figure 10As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication. The memory stores computer programs executable by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0185] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0186] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the model quantization method.

[0187] In some embodiments, the model quantization method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model quantization method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the model quantization method by any other appropriate means, such as by means of firmware.

[0188] Optionally, the model quantization method can include the following operations:

[0189] determining a current target quantization layer of a current target quantization block in the target quantization model;

[0190] inference on the target quantization model to obtain inference activation data of the current target quantization layer; wherein the inference activation data comprises inference input data and / or inference output data;

[0191] quantizing the weight parameters and the activation data of the current target quantization layer according to the model quantization configuration file to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantized weight and quantized activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; the quantized activation data comprises quantized input data and / or quantized output data;

[0192] calculating quantization error of the current target quantization layer according to the inference activation data and the quantized activation data;

[0193] in a case where it is determined that each quantization block of the target quantization model completes the quantization process, updating quantization parameters of each quantization block according to quantization error of a quantization layer in each quantization block.

[0194] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0195] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0196] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0197] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0198] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0199] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

Claims

1. A model quantization method, characterized in that, The method comprises the following steps: determining a current target quantization layer of a current target quantization block in a target quantization model; inference is performed on the target quantization model to obtain inference activation data of the current target quantization layer; wherein the inference activation data comprises inference input data and / or inference output data; quantizing the weight parameters and activation data of the current target quantization layer according to a model quantization configuration file to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantized weights and quantized activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; the quantized activation data comprises quantized input data and / or quantized output data; calculating the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data; updating the quantization parameters of each quantization block according to the quantization error of the quantization layer in each quantization block in the case that each quantization block of the target quantization model completes the quantization process; determining the target data type supported by the hardware structure of the target device running the target quantization model; compressing the quantized weights of the target quantization model according to the target data type supported by the hardware structure of the target device; loading the quantization data of the current quantization layer and the target quantization reference parameters of the target quantization model in the process of running the target quantization model on the target device; in the case that the quantized input data and the quantization result of the quantized weights are both int4, performing four-bit matrix multiplication on the quantized input data and the quantized weights, and performing product operation on the matrix multiplication result and the target quantization reference parameters to obtain the output data of the current quantization layer; in the case that the quantization result of the quantized input data is int8 and the quantization result of the quantized weights is int4, decompressing the quantized weights into int8 data type, then performing eight-bit integer matrix multiplication on the quantized input data and the decompressed quantized weights, and performing product operation on the multiplication result and the target quantization reference parameters to obtain the output data of the current quantization layer; in the case that the quantization result of the quantized input data is fp16 and the quantization result of the quantized weights is int4, decompressing the quantized weights into int8 data type, performing product operation on the decompressed quantized weights and the target quantization reference parameters, then performing matrix multiplication on the quantized input data and the quantized weights after the product operation, and taking the matrix multiplication result as the output data of the current quantization layer; taking the current quantization layer output data or the quantization result of the current quantization layer output data as the quantization input data of the next quantization layer, and outputting to the next quantization layer.

2. The method of claim 1, wherein, The quantization of the weight parameters and activation data of the current target quantization layer according to the model quantization configuration file to obtain the quantization data of the current target quantization layer comprises the following steps: determining a weight parameter quantization threshold according to the weight parameters of the current target quantization layer; calculating a first quantization scale factor of the weight parameters according to the weight parameter quantization threshold; determining a first associated quantization parameter according to the model quantization configuration file, and quantizing a weight parameter of the current target quantization layer according to the first associated quantization parameter and the first quantization scale factor, to obtain quantized weights; determining a second quantization scale factor of activation data of the current target quantization layer; determining a second associated quantization parameter according to the model quantization configuration file, and quantizing the activation data of the current target quantization layer according to the second associated quantization parameter and the second quantization scale factor, to obtain quantized activation data.

3. The method of claim 1, wherein, If the activation data includes input data and output data of the current target quantization layer, and the quantized activation data includes quantized input data and quantized output data, then the calculation of the quantization error of the current target quantization layer according to the inference activation data and the quantized activation data includes: dequantizing the quantized weights of the current target quantization layer after quantization, to obtain dequantized weights; dequantizing the quantized input data of the current target quantization layer after quantization, to obtain dequantized input data; calculating a product value of the dequantized input data and the dequantized weights, to obtain an intermediate value of output data; quantizing and dequantizing the intermediate value of output data, to obtain dequantized output data; calculating an input-output data quantization error according to the inference input-output data and the dequantized input-output data.

4. The method of claim 1, wherein, The updating of the quantization parameters of each quantization block according to the quantization errors of the quantization layers in each quantization block includes: determining a quantization error threshold; determining a quantization layer whose quantization error is greater than the quantization error threshold in each quantization block as a quantization layer to be updated; modifying the quantization configuration parameters of the quantization layer to be updated in the model quantization configuration file, and determining the quantization layer to be updated as the current target quantization layer, and returning to perform the operation of quantizing the weight parameters and the activation data of the current target quantization layer according to the model quantization configuration file, until the quantization error of the quantization layer to be updated is less than or equal to the quantization error threshold; saving the quantization parameters of the quantization layers in each quantization block; wherein the quantization parameters include a quantization scale factor of the weight parameters, a bit number of the quantized weight parameters, a quantization scale factor of the activation data, and a bit number of the quantized activation data.

5. The method of claim 1, wherein, The compression processing of the quantized weights of the target quantization model according to the target data type supported by the hardware structure of the target device includes: in a case where it is determined that the target data type supported by the hardware structure of the target device includes int4, determining a first quantized weight in a current quantized weight group to be compressed and a bit number of each quantized weight in the current quantized weight group to be compressed; in a case where it is determined that the first quantized weight is positive, performing left shift processing on the current quantized weight to be compressed according to the bit number of each quantized weight, and then performing bitwise OR processing on the current quantized weight to be compressed in combination with a previous compressed quantized weight result of the current quantized weight to be compressed, to obtain a current compressed quantized weight. In a case where the first quantization weight is determined to be negative, the first quantization weight is complemented according to the bit number of each quantization weight, and the current to-be-compressed quantization weight is left-shifted and then low-bit padded to obtain a padded quantization weight, the padded quantization weight is ANDed with a previous compressed quantization weight result of the current to-be-compressed quantization weight to obtain a current compressed quantization weight.

6. A model quantization apparatus characterized by comprising: Comprise: A current target quantization layer determination module is configured to determine a current target quantization layer of a current target quantization block in a target quantization model. An inference activation data acquisition module is configured to perform inference on the target quantization model to obtain inference activation data of the current target quantization layer. A quantization data acquisition module is configured to quantize weight parameters and activation data of the current target quantization layer according to a model quantization configuration file to obtain quantization data of the current target quantization layer; wherein the quantization data of the current target quantization layer comprises quantization weights and quantization activation data of the current target quantization layer; the activation data comprises input data and / or output data of the current target quantization layer; and the quantization activation data comprises quantized input data and / or quantized output data. A quantization error calculation module is configured to calculate quantization errors of the current target quantization layer according to the inference activation data and the quantization activation data. A quantization parameter updating module is configured to update quantization parameters of each quantization block according to quantization errors of quantization layers in each quantization block in a case where each quantization block of the target quantization model completes a quantization process. A quantization weight compression processing module is configured to determine a target data type supported by a hardware structure of a target device running the target quantization model, and to compress quantization weights of the target quantization model according to the target data type supported by the hardware structure of the target device. The model data calculation module is configured to: load quantization data and target quantization reference parameters of a current quantization layer of the target quantization model during running of the target quantization model on the target device; in a case where the quantization input data and the quantization result of the quantization weight are both int4, perform four-bit matrix multiplication on the quantization input data and the quantization weight, and perform product operation on the matrix multiplication result and the target quantization reference parameters to obtain current quantization layer output data; in a case where the quantization result of the quantization input data is int8 and the quantization result of the quantization weight is int4, decompress the quantization weight into int8 data type, perform eight-bit integer matrix multiplication on the quantization input data and the decompressed quantization weight, and perform product operation on the multiplication result and the target quantization reference parameters to obtain the current quantization layer output data; in a case where the quantization result of the quantization input data is fp16 and the quantization result of the quantization weight is int4, decompress the quantization weight into int8 data type, perform product operation on the decompressed quantization weight and the target quantization reference parameters, perform matrix multiplication on the quantization input data and the quantization weight after the product operation, and take the matrix multiplication result as the current quantization layer output data; and take the current quantization layer output data or a quantization result of the current quantization layer output data as quantization input data of a next quantization layer and output the quantization input data to the next quantization layer.

7. An electronic device, comprising: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the model quantization method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the model quantization method of any one of claims 1-5 when executed.

Images