Quantization parameter storage method and device, model reasoning method and device and storage medium

Abstract

The invention provides a quantization parameter storage method and device, a model reasoning method and device and a storage medium, and relates to the technical field of computers, in particular to the technical field of large models, the technical field of artificial intelligence and the technical field of model quantization. According to the specific implementation scheme, a calculation unit of a processor performs statistics based on reference data to obtain a statistical value of a first quantization parameter of a model; searching a target value of the first quantization parameter and a target value of a second quantization parameter of a model in a search space through the calculation unit based on the statistical value of the first quantization parameter; a target value of the first quantization parameter and a target value of the second quantization parameter are stored in a memory by the calculation unit. The quantization parameters are calculated and stored in advance, so that the quantization parameters which are calculated in advance can be read according to specific use requirements for quantization reasoning, occupation of a memory of a processor is reduced, calculation resources needing to be occupied in the reasoning process are reduced, and the reasoning speed and efficiency are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of large model technology, artificial intelligence technology, and model quantization technology. Background Technology

[0002] The cost of model inference increases significantly with the number of model parameters and the amount of context. Large models have enormous numbers of parameters and contexts. For example, some large models have tens of billions of parameters, and some have millions of words of context. Low bit quantization can reduce the memory usage of the graphics processing unit (GPU) and reduce the cost of deploying large models. Summary of the Invention

[0003] This disclosure provides a method for storing quantized parameters, a model inference method, an apparatus, and a storage medium.

[0004] According to one aspect of this disclosure, a method for storing quantization parameters is provided, comprising:

[0005] The processor's computing unit obtains the statistical values ​​of the first quantization parameter of the model based on the benchmark data.

[0006] Based on the statistical value of the first quantization parameter, the calculation unit searches the search space to obtain the target value of the first quantization parameter and the target value of the second quantization parameter of the model.

[0007] The calculation unit saves the target values ​​of the first quantization parameter and the second quantization parameter into the memory.

[0008] According to another aspect of this disclosure, a model reasoning method is provided, comprising:

[0009] The processor's computing unit processes the input data of the model to obtain a key-value matrix in the first format;

[0010] The computing unit reads the target values ​​of the first quantization parameter and the second quantization parameter from the memory, wherein the target values ​​of the first quantization parameter and the second quantization parameter are stored in the memory before the model inference process using any of the methods described above;

[0011] The first format key-value matrix is ​​quantized using a quantization function constructed by the computing unit based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain a second format key-value matrix.

[0012] The computing unit saves the second-format key-value matrix into the processor's key-value cache.

[0013] According to another aspect of this disclosure, a storage device for quantized parameters is provided, comprising:

[0014] The statistics module is used to obtain the statistical values ​​of the first quantization parameters of the model based on the benchmark data through the processor's computing unit;

[0015] The search module is used to search the search space based on the statistical value of the first quantization parameter and the target value of the second quantization parameter of the model through the calculation unit.

[0016] The calculation module is used to save the target values ​​of the first quantization parameter and the second quantization parameter into the memory through the calculation unit.

[0017] According to another aspect of this disclosure, a model inference apparatus is provided, comprising:

[0018] The processing module is used to process the input data of the model through the computing unit of the processor to obtain the key-value matrix in the first format;

[0019] The first reading module is used to read the target value of the first quantization parameter and the target value of the second quantization parameter from the memory through the computing unit, wherein the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using any of the above-mentioned means;

[0020] The quantization module is used to quantize the key-value matrix of the first format to obtain the key-value matrix of the second format by using a quantization function constructed by the computing unit based on the target value of the first quantization parameter and the target value of the second quantization parameter;

[0021] A storage module is used to save the second-format key-value matrix into the processor's key-value cache via the computing unit.

[0022] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and

[0024] The memory is communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0026] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0027] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0028] According to this disclosure, since quantization parameters can be pre-calculated and stored, and then during inference, the pre-calculated parameters can be retrieved according to specific usage requirements for quantization inference, the memory usage of the processor can be reduced. Because it is not necessary to repeatedly calculate quantization parameters during inference, the computational resources required for inference can be reduced, improving inference speed and efficiency.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0030] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0031] Figure 1 This is a schematic diagram of the structure of a large language model according to an embodiment of the present disclosure;

[0032] Figure 2 This is a schematic diagram of the structure of a multi-head attention layer according to an embodiment of the present disclosure;

[0033] Figure 3 This is a flowchart illustrating a method for storing quantization parameters according to an embodiment of the present disclosure;

[0034] Figure 4 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure;

[0035] Figure 5 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure;

[0036] Figure 6 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure.

[0037] Figure 7 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure;

[0038] Figure 8 This is a schematic flowchart of a model reasoning method according to an embodiment of the present disclosure;

[0039] Figure 9 This is a schematic flowchart of a model reasoning method according to another embodiment of the present disclosure;

[0040] Figure 10 This is a schematic diagram illustrating an application scenario of ABQ according to an embodiment of the present disclosure;

[0041] Figure 11 This is a schematic diagram of the structure of a storage device for quantization parameters according to an embodiment of the present disclosure;

[0042] Figure 12 This is a schematic diagram of the structure of a storage device for quantization parameters according to another embodiment of the present disclosure;

[0043] Figure 13 This is a schematic diagram of the structure of a model reasoning device according to an embodiment of the present disclosure;

[0044] Figure 14 This is a schematic diagram of the structure of a model inference device according to another embodiment of the present disclosure;

[0045] Figure 15 This is a block diagram of an electronic device used to implement the methods of this disclosure. Detailed Implementation

[0046] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0047] Large-scale models are increasingly being used globally. For example, some large models can solve problems related to dialogue, logical reasoning, code generation, and knowledge-based question answering. Some large models have been applied and implemented in various scenarios within the Chinese language domain. Some models have up to 70 billion parameters, while others generate data from a context of 2 million characters. For instance, during inference, a model with 70 billion parameters requires 140GB of GPU memory. If a context of 2 million characters (around 400GB) is added, the requirement increases to over 500GB of memory. Considering that some GPUs have only 80GB of memory per card, without any optimization, eight GPUs would be needed to meet the requirements, and only one user could be supported at a time, resulting in very high inference costs.

[0048] Low-bit key-value cache (KV Cache) quantization (hereinafter referred to as C4, C2) can include dynamic quantization, hybrid quantization, etc., but the above quantization methods have some problems. For example, in the dynamic C4 quantization scheme, for each query, the quantization scaling factor and other information must be calculated in each decoding process to ensure quantization accuracy. Since the quantization scale and other information need to be repeatedly counted during the inference process, it brings additional inference overhead and does not meet the actual application requirements. As another example, hybrid bit quantization of C4 and C8 requires modification of model networking and other operations to ensure inference performance. Furthermore, some non-quantization inference methods also bring additional inference time. For example, prompt compression requires compression by a small pre-model, and the pre-model requires inference time; token eviction requires specific eviction policies and must be calculated in conjunction with the token.

[0049] Figure 1 This is a schematic diagram of a model structure. The solution based on embodiments of this disclosure can propose a low-cost inference deployment scheme for large language models (LLMs). LLMs are used to solve common natural language tasks, including semantic understanding, multi-turn conversational chat, logical reasoning, coding, and text creation. Structurally, the model consists of a stack of several transformer layers. Each layer contains layer normalization (LayerNorm), multi-head attention, and fully connected layers (FeedForward). After the input text is processed by the text and positional embedding representation layers, features such as text vectors or text tensors can be obtained.

[0050] To reduce inference costs (post-training), the multi-head attention module in the above model structure needs to store key-value cache information. When the model's input information is very long (e.g., a scenario with 2 million characters), the memory and computational demands are significant, resulting in high inference costs for large models. Figure 2As shown, in a multi-head attention module architecture, the values ​​of keys (K) and values ​​(V) need to be stored during the actual inference process. Due to the repeated storage and retrieval operations, the computational bandwidth and GPU memory required are significant. For example, the Scaled Dot-Product Attention module can perform matrix multiplication (MatMul), scaling, masking, and softmax operations on the query (Q) matrix and key (K), and then perform matrix multiplication (MatMul) on the calculated result and the value (V) matrix.

[0051] Figure 3 This is a flowchart illustrating a method for storing quantization parameters according to an embodiment of the present disclosure. The method may include:

[0052] S310. The processor's computing unit obtains the statistical values ​​of the first quantization parameter of the model based on the benchmark data.

[0053] S320. The calculation unit searches the search space based on the statistical value of the first quantization parameter to obtain the target value of the first quantization parameter and the target value of the second quantization parameter of the model.

[0054] S330. The target values ​​of the first quantization parameter and the second quantization parameter are saved to the memory through the calculation unit.

[0055] In this embodiment of the disclosure, the processor may be a graphics processing unit (GPU), a central processing unit (CPU), or a neural processing unit (NPU) (or neural network processing unit), etc. The processor may include computing units, memory, cache, etc. For example, the computing units in a GPU may include stream multiprocessors (SM), and the memory may also be referred to as video memory.

[0056] In this embodiment of the disclosure, quantization can reduce the cache usage of the processor during model inference on the trained model. Various quantization parameters are required during model inference, such as the quantization scaling factor and the quantization zero point. If a dynamic quantization scheme is used, the quantization parameters need to be repeatedly calculated during inference, consuming significant computational resources. This embodiment of the disclosure can employ a static quantization scheme, which pre-calculates the quantization parameters required for inference and saves these parameters to a memory such as a hard disk.

[0057] In this embodiment of the disclosure, the statistical values ​​of the quantization parameters of the statistical model can be based on benchmark data. Benchmark data can be extracted from the training samples of the model. For example, benchmark data may include text information, see [link to relevant documentation]. Figure 1 After inputting the benchmark data into the model, the benchmark data can be embedded and encoded to obtain benchmark features, such as benchmark vectors or benchmark tensors.

[0058] In this embodiment, the model may have an attention layer, such as a self-attention layer or a multi-head self-attention layer. Statistical rules can be set in the attention layer. See also... Figure 1 After processing the baseline features through a normalization layer, the computation unit can input them into the attention layer. The attention layer can statistically analyze the received features according to statistical rules to obtain the statistical values ​​of the quantization parameters. Statistical rules can include statistical averages, the maximum of statistical absolute maximums, etc. Rules based on statistical averages can calculate the average of multiple baseline data points, and rules based on the maximum of statistical absolute maximums can calculate the maximum of the absolute maximums of multiple baseline data points. Then, based on the statistical results, the statistical value of the first quantization parameter of the model can be calculated. The statistical value of the first quantization parameter can be stored in a cache or memory. Furthermore, a pre-stored search space can be read from memory, and based on the search values ​​in the search space and the statistical value of the first quantization parameter, the target values ​​of the first and second quantization parameters of the model can be obtained. In this embodiment, the first quantization parameter can be a quantization scaling factor (scale), and the second quantization parameter can be a quantization zero point (zero_point). The computation unit can save the target values ​​of the first and second quantization parameters to a memory such as a hard disk.

[0059] In this embodiment, since quantization parameters can be pre-calculated and stored, and then during inference, the pre-calculated parameters can be read according to specific usage requirements for quantization inference, the memory usage of the processor can be reduced. Because it is not necessary to repeatedly calculate quantization parameters during inference, the computational resources required for inference can be reduced, improving inference speed and efficiency.

[0060] In one implementation, the target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during model inference and used to quantize the first format key-value matrix required by the model's attention layer into a second format key-value matrix; wherein the second format key-value matrix is ​​stored in the processor's key-value cache.

[0061] In this embodiment of the disclosure, during model inference, the processor can read the required quantization parameters from memory according to the quantization requirements of each layer of the model. For example, if it is necessary to quantize the key-value (KV) matrix of the attention layer of the model, a first quantization parameter, such as the target value of the quantization scaling factor, and a second quantization parameter, such as the target value of the quantization zero point, can be read from memory to quantize the key-value matrix required by the attention layer. For example, before quantization, the first format of the key-value matrix required by the attention layer is 16-bit Brain Floating Point 16 (BF16) format. After quantization, the second format of the key-value matrix required by the attention layer is 4-bit Integer (INT4) format. The INT4 format occupies less storage space than the BF16 format. The key-value matrix required by the attention layer can be stored in the processor's key-value cache (KV cache) after quantization. Since the quantized KV matrix occupies less storage space than the unquantized KV matrix, and the key-value cache is usually in the processor's memory, the memory usage of the processor, such as the GPU's video memory, can be reduced. Since quantization can be performed using pre-calculated and stored KV matrix quantization parameters, it is not necessary to calculate the quantization parameters first and then perform quantization. This reduces the computational resources required for the quantization process during inference, thereby improving inference speed and efficiency.

[0062] In one implementation, the target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during model inference and used to dequantize the second-format key-value matrix read from the key-value cache into a first-format key-value matrix; wherein the dequantized key-value matrix is ​​used as the input feature of the attention layer.

[0063] For example, if dequantization is needed on a key-value (KV) matrix in a second format from the key-value cache, the processor can read the target value of a first quantization parameter (e.g., a quantization scaling factor) and a second quantization parameter (e.g., a quantization zero-point target value) from memory to dequantize the key-value matrix required by the attention layer. The first quantization parameter can also be the dequantization scaling factor, or the dequantization scaling factor can be derived from the quantization scaling factor. For example, dequantization can convert a key-value matrix in a second format back to a key-value matrix in a first format. After dequantization, the accuracy of the key-value matrix input to the attention layer of the model can be improved. Since the quantized KV matrix in memory can be dequantized using pre-computed and stored quantization parameters, and since it is not necessary to compute the quantization parameters before dequantization, the computational resources required for the dequantization process during inference can be reduced, improving inference speed and efficiency.

[0064] Figure 4 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, the processor's computing unit statistically obtains statistical values ​​of the model's first quantization parameter based on reference data, including at least one of the following:

[0065] S410. Calculate the average minimum and average maximum values ​​of the first quantization parameter based on the average value of the benchmark data;

[0066] S420. Calculate the minimum and maximum values ​​of the absolute maximum values ​​of the first quantization parameter based on the absolute maximum values ​​of the benchmark data.

[0067] In this embodiment, the timing of S410 and S420 is not limited. S410 can be executed first and then S420, or S420 can be executed first and then S410, or only one of the steps can be executed.

[0068] For example, if a set of baseline data corresponds to a set of features, the average and the maximum of the absolute maximum of that set of features can be calculated first. Using the rule of statistical averaging, the minimum and maximum average values ​​can be obtained from the average values ​​corresponding to N baseline data. Using the rule of statistical maximum of absolute maximum values, the minimum and maximum absolute values ​​can be obtained from the absolute values ​​corresponding to N baseline data. Based on one or more of the minimum average, the minimum of the absolute maximum, the maximum average, and the maximum of the absolute maximum, the target value of the first quantization parameter can be searched more accurately.

[0069] Figure 5This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, based on the statistical value of the first quantization parameter, a target value for the first quantization parameter and a target value for the second quantization parameter of the model are searched in the search space to obtain the target value of the first quantization parameter and the target value of the model's second quantization parameter, including:

[0070] S510. Based on the minimum of the average minimum, the minimum of the absolute maximum, the maximum of the average maximum, the maximum of the absolute maximum, and the search parameters in the search space, calculate the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter.

[0071] S520. Based on the candidate values ​​of the first quantization parameter and the second quantization parameter corresponding to all search parameters in the search space, and the key value matrix of the benchmark data, calculate the value of the loss function respectively, so as to search for the target search parameter that minimizes the value of the loss function.

[0072] S530. Based on the target search parameters, calculate the target value of the first quantization parameter and the target value of the second quantization parameter.

[0073] In this embodiment, the search parameters may include multiple search values. One or more search spaces can be pre-set in the processor's memory. The computing unit can read the search values ​​from the search space in memory for subsequent calculations. For example, the search space S = [0, 0.2, 0.4, 0.6]. When calculating the candidate values ​​of the first quantization parameter and the second quantization parameter, the computing unit can use the same search parameters. For example, a search parameter s, such as 0.2, can be selected from the search space S and substituted into the relevant formula for the quantization parameter to calculate the candidate values ​​of the first and second quantization parameters. Then, the candidate values ​​of the first and second quantization parameters are substituted into the formula for the loss function to calculate the loss value corresponding to s. By comparing all the loss values ​​corresponding to s in the search space, the s with the smallest loss value is obtained. o This refers to the target search parameters. Then, the target search parameters s... o By substituting the relevant formulas for the quantization parameters, the target values ​​of the first quantization parameter and the second quantization parameter can be calculated.

[0074] In this embodiment, the target value of the quantization parameter can be quickly obtained based on the search space, improving computational speed and efficiency. Furthermore, the search parameters in the search space can be optimized based on the search process. For example, if the loss value corresponding to a search parameter is large, such as exceeding a threshold, the search parameter can be deleted. Similarly, if a larger search parameter corresponds to a larger loss value, more search parameters with smaller values ​​can be added. Conversely, if a larger search parameter corresponds to a smaller loss value, more search parameters with larger values ​​can be added. Optimizing the search space further improves search speed and efficiency.

[0075] Figure 6 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, calculating candidate values ​​for the first quantization parameter and the second quantization parameter based on the average minimum, the minimum of the absolute maximum, the average maximum, the maximum of the absolute maximum, and search parameters in the search space includes:

[0076] S610. Based on the minimum of the average, the minimum of the absolute maximum, and the search parameters in the search space, calculate the minimum value of the first quantization parameter.

[0077] S620. Based on the average maximum value, the maximum value of the absolute maximum value, and the search parameters in the search space, calculate the maximum value of the first quantization parameter;

[0078] S630. Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter;

[0079] S640. Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, calculate the candidate value of the second quantization parameter.

[0080] An example of a formula for the minimum value of the first quantization parameter is as follows:

[0081] scale_min=s*avg_min+(1-s)*absmax_min Formula 1

[0082] Here, `scale_min` represents the minimum value of the first quantization parameter, `avg_min` represents the minimum average value, and `absmax_min` represents the minimum absolute maximum value. `avg` represents the average value after feature normalization, and `absmax` represents the absolute maximum value after feature normalization. The minimum average value (`avg_min`) can be obtained by statistically analyzing the `avg` values ​​of multiple benchmark data points; similarly, the minimum absolute maximum value (`absmax_min`) can be obtained by statistically analyzing the `absmax` values ​​of multiple benchmark data points. `s` can represent the search parameters in the search space `S`, where `s` belongs to `S`.

[0083] An example of a formula for the maximum value of the first quantization parameter is as follows:

[0084] scale_max=s*avg_max+(1-s)*absmax_max Formula 2

[0085] Where scale_max represents the maximum value of the first quantization parameter, avg_max represents the average maximum value, and absmax_max represents the maximum absolute maximum value. The largest average value, avg_max, can be obtained by statistically analyzing the avg values ​​of multiple benchmark data points; similarly, the largest absolute maximum value, absmax_max, can be obtained by statistically analyzing the absmax values ​​of multiple benchmark data points. The meaning of 's' is the same as in Formula 1, and the search parameters in Formulas 1 and 2 can take the same value.

[0086] An example of a formula for a candidate value of the first quantization parameter is as follows:

[0087]

[0088] The meanings of scale_max and scale_min are given in Formulas 1 and 2, respectively. They can be calculated using Formulas 1 and 2, and this disclosure does not restrict the calculation order of Formulas 1 and 2. Scale can represent the candidate value of the first quantization parameter. According to Formula 3, multiple candidate values ​​of the first quantization parameter corresponding to multiple search parameters can be calculated.

[0089] An example of a formula for a candidate value of the second quantization parameter is as follows:

[0090]

[0091] Here, `scale_min` can be obtained through formula 1, and the meaning of `scale` is given in formula 3. `round()` performs rounding, and `clip()` retrieves a value from the set of values ​​within parentheses that does not exceed the upper or lower boundaries. The second element within the parentheses represents the lower boundary of the value, and the third element represents the upper boundary. That is, if the value of the first element does not exceed the upper or lower boundaries, the value of the first element is taken as the calculation result; if it is less than the lower boundary, the value of the lower boundary (the second element) is taken as the calculation result; if it is greater than the upper boundary, the value of the upper boundary (the third element) is taken as the calculation result.

[0092] For example, substituting the average minimum value (avg_min), the minimum absolute maximum value (absmax_min) obtained from the above statistics, and a search parameter s selected in the search space into Formula 1, we can calculate the minimum value of the quantization scaling factor, scale_min. Substituting the average maximum value (avg_max), the maximum absolute maximum value (absmax_max) obtained from the above statistics, and a search parameter s selected in the search space into Formula 2, we can calculate the maximum value of the quantization scaling factor, scale_max. Substituting scale_min and scale_max into Formula 3 for the candidate values ​​of the first quantization parameter, we can obtain the candidate value of the first quantization parameter, scale. Then, substituting scale_min and scale into Formula 4 for the candidate values ​​of the second quantization parameter, we can obtain the candidate value of the second quantization parameter, zero_point.

[0093] In this embodiment, the candidate values ​​of the quantization parameters calculated based on multiple statistical values ​​of the quantization parameters are more accurate. Using the same search parameters within the same search space can improve search speed.

[0094] Figure 7 This is a flowchart illustrating a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, calculating candidate values ​​for the first quantization parameter and the second quantization parameter based on the average minimum, the minimum of the absolute maximum, the average maximum, the maximum of the absolute maximum, and search parameters in the search space includes:

[0095] S710. Based on the minimum of the average, the minimum of the absolute maximum, and the first search parameter in the first search space, calculate the minimum value of the first quantization parameter.

[0096] S720. Based on the maximum value of the average maximum value, the maximum value of the absolute maximum value, and the second search parameter in the second search space, calculate the maximum value of the first quantization parameter;

[0097] S730. Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter;

[0098] S740. Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, calculate the candidate value of the second quantization parameter.

[0099] An example of a formula for the minimum value of the first quantization parameter is as follows:

[0100] scale_min=s1*avg_min+(1-s1)*absmax_min Formula 5

[0101] The meanings of scale_min, avg_min, and absmax_min can be found in Equation 1, and will not be repeated here. s1 is the first search parameter belonging to the first search space S1.

[0102] An example of a formula for the maximum value of the first quantization parameter is as follows:

[0103] scale_max = s * *avg_max+(1-s * )*absmax_max formula 6

[0104] The meanings of scale_max, avg_max, and absmax_max are the same as in Formula 2, and will not be repeated here. s2 belongs to the second search space, S2 is the second search parameter.

[0105] In this embodiment, s1 and s2 can be different search parameters in different search spaces. In one implementation, s1 and s2 can also be different search parameters in the same search space, in which case the first search space and the second search space can be understood as the same search space.

[0106] After calculating scale_min and scale_max based on formulas 5 and 6, scale and zero_point can be calculated by referring to formulas 4 and 5.

[0107] For example, substituting the average minimum value avg_min, the minimum absolute maximum value absmax_min obtained from the above statistics, and a certain first search parameter s1 selected in the first search space S1, into Formula 5 above, the minimum value of the quantization scaling factor scale_min can be calculated. Substituting the average maximum value avg_max, the maximum absolute maximum value absmax_max obtained from the above statistics, and a certain second search parameter s2 selected in the second search space S2, into Formula 6 above, the maximum value of the quantization scaling factor scale_max can be calculated. Substituting scale_min and scale_max into Formula 3 for the candidate values ​​of the first quantization parameter, the candidate value scale of the first quantization parameter can be obtained. Then, substituting scale_min and scale into Formula 4 for the candidate values ​​of the second quantization parameter, the candidate value zero_point of the second quantization parameter can be obtained.

[0108] In this embodiment of the disclosure, the candidate values ​​of the quantization parameter calculated based on multiple statistical values ​​of the quantization parameter are more accurate. Using different search parameters can improve the accuracy of the search results.

[0109] In one implementation, the loss function is determined based on the quantization function and the inverse quantization function.

[0110] In one implementation, the quantization function is used to perform a rounding operation on the first format key-value matrix of the reference data based on the target values ​​of the first quantization parameter and the second quantization parameter, to obtain a quantized second format key-value matrix.

[0111] An example of a quantization function is as follows:

[0112]

[0113] Where Q represents the quantization function, quant_x represents the second-format key-value matrix after quantizing a reference data x, and int() converts the data in parentheses into 4-bit integers. The explanations of parameters such as clip(), scale, and zero_point are similar to those above and can be found in the relevant descriptions above, so they will not be repeated here.

[0114] In one implementation, the dequantization function is used to perform floating-point operations on the quantized second format key-value matrix based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the dequantized first format key-value matrix.

[0115] An example of an inverse quantization function is as follows:

[0116] DQ:dequant_quant_x=bfloat16((quant_x-zero_point)*scale) Formula 8

[0117] Here, DQ represents the dequantization function, dequant_quant_x indicates that a quantized second-format key-value matrix is ​​dequantized to the base data x, and bfloat16() indicates that the format of the data in parentheses is converted to 16-bit floating-point data. Furthermore, the explanations of parameters such as scale, zero_point, and quant_x are similar to those above and can be found in the relevant descriptions above; they will not be repeated here.

[0118] An example of a loss function formula is as follows:

[0119]

[0120] Where x represents the tensor corresponding to a baseline data point (e.g., input text), Q can be found in Equation 7, and DQ can be found in Equation 8. For K and V in each transformer layer, an optimal configuration s is adaptively searched from the parameter space S to minimize the loss of KV before and after INT4 quantization. Different models will modify S based on posterior information. The loss value is calculated using the Mean Squared Error (MSE Loss) function or the top K loss function.

[0121] For example, substituting the candidate values ​​of the first and second quantization factors calculated using formulas 3 and 4, along with the current benchmark data x, into formula 7, we can quantize the second-format key-value matrix quant_x of the current benchmark data x. Substituting the candidate values ​​of the first and second quantization factors calculated using formulas 3 and 4, along with the current second-format key-value matrix quant_x, into formula 8, we can quantize the current first-format benchmark data dequant_quant_x. Substituting quant_x and dequant_quant_x into formula 9 of the loss function, we can obtain the loss value under the current search parameters and adaptively search for an optimal configuration s that minimizes the loss value, thus obtaining the search parameters corresponding to the minimum loss value and the corresponding target values ​​of the first and second quantization factors.

[0122] According to embodiments of this disclosure, a loss function can be constructed based on the multiplication result of the quantization function and the dequantization function. The resulting loss function better reflects the actual situation of information loss during quantization and dequantization of the benchmark data under the current search parameters, considering the quantization scaling factor and quantization zeros. Based on this loss function, the most suitable quantization scaling factor and quantization zeros for quantizing and dequantizing the current benchmark data can be obtained. This improves the accuracy of model inference while reducing the memory (e.g., GPU memory) usage during model inference.

[0123] Figure 8 This is a flowchart illustrating a model inference method according to an embodiment of the present disclosure. The method may include:

[0124] S810: The processor's computing unit processes the input data of the model to obtain a key-value matrix in the first format;

[0125] S820. The target value of the first quantization parameter and the target value of the second quantization parameter are read from the memory by the computing unit, wherein the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using an embodiment of any of the above-described quantization parameter storage methods.

[0126] S830. The first format key-value matrix is ​​quantized using the quantization function constructed by the computing unit based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the second format key-value matrix.

[0127] S840: The computing unit saves the second-format key-value matrix into the processor's key-value cache.

[0128] In this embodiment of the disclosure, the processor may be a graphics processing unit (GPU), a central processing unit (CPU), or a neural processing unit (NPU) (or neural network processing unit), etc. The processor may include computing units, memory, cache, etc. For example, the computing units in a GPU may include stream multiprocessors (SM), and the memory may also be called video memory. The storage may be a hard disk, such as a solid-state drive (SSD) or a hard disk drive (HDD), etc.

[0129] In this embodiment, the processor can extract features from the model's input data, such as text, to obtain feature data. The feature data is in a first format, such as a BF16 key-value matrix. Based on the current BF16 feature data, the corresponding target values ​​of the first and second quantization parameters can be read from the memory. The target values ​​of the first and second quantization parameters can be quantization parameters calculated in advance using the aforementioned quantization parameter storage method on baseline data before model inference (or during model deployment) and stored in the memory. The baseline data can be data selected from the sample data.

[0130] In this embodiment, the key-value cache can be stored in the processor's memory. Based on the target values ​​of a first quantization parameter and a second quantization parameter, the key-value matrix in a first format can be quantized to obtain a key-value matrix in a second format, such as int4. This second-format key-value matrix has a smaller data size than the first-format key-value matrix. The quantized second-format key-value matrix can be stored in the key-value cache of the processor, such as the GPU's memory (video memory).

[0131] According to embodiments of this disclosure, the baseline model to be processed can be quantized into a smaller key-value matrix, such as int4, and stored in the memory of the computing unit. Compared to storing a key-value matrix in a first format, such as BF16, this requires less storage space and saves memory resources. With the same memory resources, more users can be supported. Quantization is performed by reading pre-obtained quantization parameters from static storage, which improves the efficiency and applicability of quantization compared to calculating the quantization parameters manually.

[0132] Figure 9 This is a flowchart illustrating a model inference method according to another embodiment of the present disclosure. This embodiment may include one or more features from the above embodiments. In one implementation, the method further includes:

[0133] S910. The second-format key-value matrix is ​​read from the key-value cache through the computing unit;

[0134] S920. The inverse quantization function constructed by the computing unit based on the target values ​​of the first quantization parameter and the second quantization parameter is used to inverse quantize the second format key-value matrix to obtain the first format key-value matrix, which is then input into the attention layer of the model.

[0135] In this embodiment, when the model's attention layer needs input data for inference, it can first extract a key-value matrix in a second format from the key-value cache in memory, and then perform dequantization using the target values ​​of the corresponding first quantization parameter and the target values ​​of the second quantization parameter to obtain a key-value matrix in a first format. This first-format key-value matrix is ​​then input into the model's attention layer for inference.

[0136] According to embodiments of this disclosure, a key-value matrix in a second format can be extracted from the key-value cache and dequantized into a key-value matrix in a first format before quantization. This reduces the memory footprint of the key-value cache and improves the accuracy of modular inference.

[0137] In one implementation, the quantization function is used to perform a rounding operation on the key-value matrix of the first format based on the target value of the first quantization parameter and the target value of the second quantization parameter, so as to obtain the quantized key-value matrix of the second format.

[0138] In one implementation, the dequantization function is used to perform floating-point operations on the second format key-value matrix based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the dequantized first format key-value matrix.

[0139] In this embodiment of the disclosure, the data in the first format can be the data required for inference by inputting into the model's attention layer, and is the data required for model inference; the data in the second format can be the data stored in the memory of the computing unit, and the data in the second format can be obtained based on the data in the first format, and its memory resource occupation is less than that of the data in the first format.

[0140] In this embodiment of the disclosure, when it is necessary to use the memory of the computing unit to store the key-value matrix, the model in the first format can be quantized into data in the second format based on a quantization method. The quantization can be based on the target value of the first quantization parameter and the target value of the second quantization parameter. When it is necessary to use the model for inference, the data in the second format can be dequantized back into data in the first format. The dequantization can be based on the target value of the first quantization parameter used during the quantization of the data in the first format and the target value of the second quantization parameter.

[0141] According to embodiments of this disclosure, data in a second format that consumes less memory resources can be stored in the memory of the computing unit. During model inference, the data in the second format is read from memory and dequantized into data in the first format and input into the attention layer of the model. This reduces the memory consumption of the key-value cache and improves the accuracy of model inference.

[0142] The model inference method of this disclosure may include a static 4-bit key-value cache (KV cache) quantization scheme for large model inference, used to reduce memory consumption during inference and improve inference speed. For example, by reducing the KV cache from 16 bits to 4 bits, the memory overhead of 400GB can be reduced to 100GB. Under the premise of the same GPU resource consumption, the model can be optimized from supporting 1 user to supporting 4 users simultaneously with the same amount of resources, increasing the query-per-second (QPS) by 3 times. Applying this scheme to large model inference optimization scenarios can not only make the performance of large models nearly lossless, but also reduce inference memory overhead, improve QPS, and reduce the deployment cost of large model inference.

[0143] In one application scenario, storing the key (K) and value (V) in the model as 4-bit integers (INT4) instead of 16-bit brain floating-point (BF16) data types can accelerate the memory access process for K and V, reduce inference memory usage (-75%), significantly improve the QPS of large models, and save inference deployment costs. To achieve efficient and accurate INT4 storage, an adaptive basting quantization (ABQ) algorithm can be used, which may include two rounds of calibration.

[0144] Round 1: On the calibration data (which can be extracted from the training samples), perform statistical analysis according to the average value (avg) and the absolute maximum value (absmax) of the quantization scaling factor, and calculate the minimum value (scale_min) and the maximum value (scale_max) of the quantization scaling factor.

[0145] The second round involves searching the custom search space for the quantization scaling factor of K and V in each layer that minimizes the loss function, and calculating the corresponding quantization zero point.

[0146] In one implementation, the same s can be used for scale_min and scale_max in a single calculation. In another implementation, different s can be used for scale_min and scale_max in a single calculation.

[0147] For multiple data sets, one or more operations can be performed on the Loss of different samples under the same K or V to calculate the mean, minimum, and maximum values, and the optimal scale and zero point can be selected.

[0148] The embodiments disclosed herein can be widely applied in the field of large model inference deployment, such as general model acceleration and long text model deployment, to reduce inference costs and significantly improve QPS. One application scenario is as follows: Figure 10 As shown in the diagram. The up and down arrows represent data flow, and the left and right arrows represent model flow. During model training, the model receives training data and produces a raw model of BF16 (bfloat16) data type. This model has high inference and deployment costs. After the ABQ process, the model's KVcache data type changes from BF16 to INT4. During inference, the C4 model produced by ABQ can be repeatedly called to provide answers to user questions.

[0149] The static C4 quantization scheme (including the ABQ process) of this disclosure achieves C4 quantization without introducing additional inference overhead (e.g., time consumption), saving memory usage, improving QPS, and reducing inference costs, for example, to 25% of the original cost. In large-model products, C4 quantization is an important acceleration method, saving more than twice the cost without affecting user experience. Considering the extremely large number of users of large models, this scheme has significant cost savings potential, effectively supporting low-cost applications of large models.

[0150] Figure 11 This is a schematic diagram of a storage device for quantized parameters according to an embodiment of the present disclosure. In one embodiment, the device may include:

[0151] The statistics module 1110 is used to obtain the statistical values ​​of the first quantization parameters of the model based on the benchmark data through the processor's computing unit;

[0152] The search module 1120 is used to search the search space based on the statistical value of the first quantization parameter by the calculation unit to obtain the target value of the first quantization parameter and the target value of the second quantization parameter of the model.

[0153] The calculation module 1130 is used to save the target values ​​of the first quantization parameter and the second quantization parameter into the memory through the calculation unit.

[0154] In one implementation, the target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during model inference and used to quantize the first format key-value matrix required by the model's attention layer into a second format key-value matrix; wherein the second format key-value matrix is ​​stored in the processor's key-value cache.

[0155] In one implementation, the target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during model inference and used to dequantize the second-format key-value matrix read from the key-value cache into a first-format key-value matrix; wherein the dequantized key-value matrix is ​​used as the input feature of the attention layer.

[0156] In one implementation, the statistics module 1110 is further used for at least one of the following:

[0157] The average minimum and average maximum values ​​of the first quantization parameter are obtained by statistically analyzing the benchmark data using the average value.

[0158] The minimum and maximum values ​​of the absolute maximum values ​​of the first quantization parameter are obtained by statistically analyzing the absolute maximum values ​​of the benchmark data.

[0159] Figure 12 This is a schematic diagram of a storage device for quantization parameters according to another embodiment of the present disclosure. The device may include one or more features of the aforementioned storage device for quantization parameters. In one embodiment, the search module 1120 includes:

[0160] The first calculation submodule 1121 is used to calculate the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter based on the minimum of the average minimum value, the minimum of the absolute maximum value, the maximum of the average maximum value, the maximum of the absolute maximum value, and the search parameters in the search space.

[0161] Search submodule 1122 is used to calculate the value of the loss function based on the candidate values ​​of the first quantization parameter and the second quantization parameter corresponding to all search parameters in the search space, as well as the key value matrix of the benchmark data, so as to search for the target search parameter that minimizes the value of the loss function.

[0162] The second calculation submodule 1123 is used to calculate the target value of the first quantization parameter and the target value of the second quantization parameter based on the target search parameter.

[0163] In one embodiment, the first computing submodule 1121 is further configured to:

[0164] Based on the minimum of the average, the minimum of the absolute maximum, and the search parameters in the search space, calculate the minimum value of the first quantization parameter;

[0165] Based on the average maximum value, the maximum value of the absolute maximum value, and the search parameters in the search space, calculate the maximum value of the first quantization parameter;

[0166] Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter;

[0167] Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

[0168] In one embodiment, the first computing submodule 1121 is further configured to:

[0169] Based on the minimum of the average, the minimum of the absolute maximum, and the first search parameter in the first search space, calculate the minimum value of the first quantization parameter;

[0170] The maximum value of the first quantization parameter is calculated based on the maximum value of the average maximum value, the maximum value of the absolute maximum value, and the second search parameter in the second search space;

[0171] Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter;

[0172] Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

[0173] In one implementation, the loss function is determined based on a quantization function and an inverse quantization function;

[0174] The quantization function is used to perform a rounding operation on the first format key-value matrix of the reference data based on the target value of the first quantization parameter and the target value of the second quantization parameter, so as to obtain the quantized second format key-value matrix.

[0175] The dequantization function is used to perform floating-point operations on the quantized second format key-value matrix based on the target values ​​of the first quantization parameter and the second quantization parameter, to obtain the dequantized first format key-value matrix.

[0176] Figure 13 This is a schematic diagram of a model inference apparatus according to an embodiment of the present disclosure. In one embodiment, the apparatus may include:

[0177] The processing module 1310 is used to process the input data of the model through the computing unit of the processor to obtain a key-value matrix in the first format;

[0178] The first reading module 1320 is used to read the target value of the first quantization parameter and the target value of the second quantization parameter from the memory through the computing unit, wherein the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using an embodiment of the storage device for any of the quantization parameters described above.

[0179] The quantization module 1330 is used to quantize the key-value matrix of the first format to obtain the key-value matrix of the second format by means of a quantization function constructed by the computing unit based on the target value of the first quantization parameter and the target value of the second quantization parameter.

[0180] Storage module 1340 is used to save the second format key-value matrix into the processor's key-value cache through the computing unit.

[0181] Figure 14 This is a schematic diagram of a model inference apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the model inference apparatus described above. In one embodiment, the apparatus further includes:

[0182] The second reading module 1410 is used to read the second format key-value matrix from the key-value cache through the calculation unit;

[0183] The dequantization module 1420 is used to dequantize the second format key-value matrix to obtain the first format key-value matrix by using the dequantization function constructed by the computing unit based on the target value of the first quantization parameter and the target value of the second quantization parameter, and then input it into the attention layer of the model.

[0184] In one implementation, the quantization function is used to perform a rounding operation on the key-value matrix of the first format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the quantized key-value matrix of the second format.

[0185] The dequantization function is used to perform floating-point operations on the key-value matrix of the second format based on the target values ​​of the first quantization parameter and the second quantization parameter, so as to obtain the dequantized key-value matrix of the first format.

[0186] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0187] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0188] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0189] Figure 15A schematic block diagram of an example electronic device 1500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0190] like Figure 15 As shown, device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1502 or a computer program loaded from storage unit 1508 into random access memory (RAM) 1503. The RAM 1503 may also store various programs and data required for the operation of device 1500. The computing unit 1501, ROM 1502, and RAM 1503 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0191] Multiple components in device 1500 are connected to I / O interface 1505, including: input unit 1506, such as keyboard, mouse, etc.; output unit 1507, such as various types of monitors, speakers, etc.; storage unit 1508, such as disk, optical disk, etc.; and communication unit 1509, such as network card, modem, wireless transceiver, etc. Communication unit 1509 allows device 1500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0192] The computing unit 1501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as methods for storing quantization parameters and / or methods for model inference. For example, in some embodiments, methods for storing quantization parameters and / or methods for model inference may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1500 via ROM 1502 and / or communication unit 1509. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, one or more steps of the methods for storing quantization parameters and / or methods for model inference described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured by any other suitable means (e.g., by means of firmware) to perform a storage method for quantization parameters and / or a model inference method.

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0195] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0197] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0198] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0199] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0200] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for storing quantization parameters, comprising: The processor's computing unit obtains the statistical values ​​of the first quantization parameter of the model based on the benchmark data. The calculation unit searches the search space based on the statistical value of the first quantization parameter to obtain the target value of the first quantization parameter and the target value of the second quantization parameter of the model. The calculation unit saves the target values ​​of the first quantization parameter and the second quantization parameter into the memory.

2. The method according to claim 1, wherein, The target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during the model inference process and used to quantize the first format key-value matrix required by the model's attention layer into a second format key-value matrix; wherein the second format key-value matrix is ​​stored in the processor's key-value cache.

3. The method according to claim 2, wherein, The target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during the model inference process, and used to dequantize the second format key-value matrix read from the key-value cache into a first format key-value matrix; wherein, the dequantized key-value matrix is ​​used as the input feature of the attention layer.

4. The method according to claim 1, wherein, The processor's computing unit obtains statistical values ​​for the first quantization parameter of the model based on benchmark data, including at least one of the following: The average minimum and average maximum values ​​of the first quantization parameter are obtained by statistically analyzing the benchmark data according to the average value. The minimum and maximum values ​​of the absolute maximum values ​​of the first quantization parameter are obtained by statistically analyzing the absolute maximum values ​​of the reference data.

5. The method according to claim 4, wherein, Based on the statistical value of the first quantization parameter, the target value of the first quantization parameter and the target value of the second quantization parameter of the model are obtained by searching the search space, including: Based on the minimum of the average minimum, the minimum of the absolute maximum, the maximum of the average maximum, the maximum of the absolute maximum, and the search parameters in the search space, calculate the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter. Based on the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter corresponding to all search parameters in the search space, and the key value matrix of the benchmark data, the value of the loss function is calculated respectively to search for the target search parameter that minimizes the value of the loss function. Based on the target search parameters, calculate the target value of the first quantization parameter and the target value of the second quantization parameter.

6. The method according to claim 5, wherein, Based on the minimum of the average minimum, the minimum of the absolute maximum, the maximum of the average maximum, the maximum of the absolute maximum, and the search parameters in the search space, candidate values ​​for the first quantization parameter and candidate values ​​for the second quantization parameter are calculated, including: The minimum value of the first quantization parameter is calculated based on the minimum average value, the minimum absolute maximum value, and the search parameters in the search space. The maximum value of the first quantization parameter is calculated based on the average maximum value, the maximum value of the absolute maximum value, and the search parameters in the search space; Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter; Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

7. The method according to claim 5, wherein, Based on the minimum of the average minimum, the minimum of the absolute maximum, the maximum of the average maximum, the maximum of the absolute maximum, and the search parameters in the search space, candidate values ​​for the first quantization parameter and candidate values ​​for the second quantization parameter are calculated, including: The minimum value of the first quantization parameter is calculated based on the minimum average value, the minimum absolute maximum value, and the first search parameter in the first search space. The maximum value of the first quantization parameter is calculated based on the maximum value of the average maximum value, the maximum value of the absolute maximum value, and the second search parameter in the second search space; Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter; Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

8. The method according to any one of claims 5 to 7, wherein, The loss function is determined based on the quantization function and the inverse quantization function; The quantization function is used to perform a rounding operation on the first format key-value matrix of the reference data based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the quantized second format key-value matrix. The dequantization function is used to perform floating-point operations on the quantized second-format key-value matrix based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the dequantized first-format key-value matrix.

9. A model reasoning method, comprising: The processor's computing unit processes the input data of the model to obtain a key-value matrix in the first format; The computing unit reads the target value of the first quantization parameter and the target value of the second quantization parameter from the memory, wherein the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using the method described in any one of claims 1 to 8; The first format key-value matrix is ​​quantized to obtain the second format key-value matrix by means of the quantization function constructed by the calculation unit based on the target values ​​of the first quantization parameter and the second quantization parameter; The computing unit saves the key-value matrix in the second format into the key-value cache of the processor.

10. The method according to claim 9, wherein, The method further includes: The calculation unit reads the key-value matrix in the second format from the key-value cache; The key-value matrix in the second format is dequantized using the dequantization function constructed by the computing unit based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the key-value matrix in the first format, which is then input into the attention layer of the model.

11. The method according to claim 10, wherein, The quantization function is used to perform a rounding operation on the key-value matrix of the first format based on the target values ​​of the first quantization parameter and the target values ​​of the second quantization parameter, so as to obtain the quantized key-value matrix of the second format. The dequantization function is used to perform floating-point operations on the key-value matrix of the second format based on the target values ​​of the first quantization parameter and the second quantization parameter, to obtain the dequantized key-value matrix of the first format.

12. A storage device for quantized parameters, comprising: The statistics module is used to obtain the statistical values ​​of the first quantization parameters of the model based on the benchmark data through the processor's computing unit; The search module is used to search the search space using the calculation unit based on the statistical value of the first quantization parameter to obtain the target value of the first quantization parameter and the target value of the second quantization parameter of the model. The calculation module is used to save the target values ​​of the first quantization parameter and the second quantization parameter into the memory through the calculation unit.

13. The apparatus according to claim 12, wherein, The target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during the model inference process and used to quantize the first format key-value matrix required by the model's attention layer into a second format key-value matrix; wherein the second format key-value matrix is ​​stored in the processor's key-value cache.

14. The apparatus according to claim 13, wherein, The target values ​​of the first quantization parameter and the second quantization parameter can be read from the memory into the processor during the model inference process, and used to dequantize the second format key-value matrix read from the key-value cache into a first format key-value matrix; wherein, the dequantized key-value matrix is ​​used as the input feature of the attention layer.

15. The apparatus according to claim 12, wherein, The statistics module is also used for at least one of the following: The average minimum and average maximum values ​​of the first quantization parameter are obtained by statistically analyzing the benchmark data according to the average value. The minimum and maximum values ​​of the absolute maximum values ​​of the first quantization parameter are obtained by statistically analyzing the absolute maximum values ​​of the reference data.

16. The apparatus according to claim 15, wherein, The search module includes: The first calculation submodule is used to calculate the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter based on the minimum of the average minimum, the minimum of the absolute maximum, the maximum of the average maximum, the maximum of the absolute maximum, and the search parameters in the search space. The search submodule is used to calculate the value of the loss function based on the candidate values ​​of the first quantization parameter and the second quantization parameter corresponding to all search parameters in the search space, as well as the key value matrix of the benchmark data, so as to search for the target search parameter that minimizes the value of the loss function. The second calculation submodule is used to calculate the target value of the first quantization parameter and the target value of the second quantization parameter based on the target search parameters.

17. The apparatus according to claim 16, wherein, The first calculation submodule is further configured to: The minimum value of the first quantization parameter is calculated based on the minimum average value, the minimum absolute maximum value, and the search parameters in the search space. The maximum value of the first quantization parameter is calculated based on the average maximum value, the maximum value of the absolute maximum value, and the search parameters in the search space; Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter; Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

18. The apparatus according to claim 16, wherein, The first calculation submodule is further configured to: The minimum value of the first quantization parameter is calculated based on the minimum average value, the minimum absolute maximum value, and the first search parameter in the first search space. The maximum value of the first quantization parameter is calculated based on the maximum value of the average maximum value, the maximum value of the absolute maximum value, and the second search parameter in the second search space; Based on the maximum and minimum values ​​of the first quantization parameter, calculate the candidate values ​​of the first quantization parameter; Based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter, the candidate value of the second quantization parameter is calculated.

19. The apparatus according to any one of claims 16 to 18, wherein, The loss function is determined based on the quantization function and the inverse quantization function; The quantization function is used to perform a rounding operation on the first format key-value matrix of the reference data based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the quantized second format key-value matrix. The dequantization function is used to perform floating-point operations on the quantized second-format key-value matrix based on the target values ​​of the first quantization parameter and the second quantization parameter to obtain the dequantized first-format key-value matrix.

20. A model reasoning device, comprising: The processing module is used to process the input data of the model through the computing unit of the processor to obtain the key-value matrix in the first format; The first reading module is configured to read the target value of the first quantization parameter and the target value of the second quantization parameter from the memory through the computing unit, wherein the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using the means as described in any one of claims 1 to 7; The quantization module is used to quantize the key-value matrix in the first format to obtain the key-value matrix in the second format by using a quantization function constructed by the calculation unit based on the target values ​​of the first quantization parameter and the second quantization parameter. A storage module is used to save the key-value matrix of the second format to the key-value cache of the processor through the computing unit.

21. The apparatus according to claim 20, wherein, The device further includes: The second reading module is used to read the key-value matrix of the second format from the key-value cache through the calculation unit; The dequantization module is used to dequantize the key-value matrix in the second format using a dequantization function constructed by the computing unit based on the target values ​​of the first quantization parameter and the second quantization parameter, to obtain a key-value matrix in the first format, which is then input into the attention layer of the model.

22. The apparatus according to claim 21, wherein, The quantization function is used to perform a rounding operation on the key-value matrix of the first format based on the target values ​​of the first quantization parameter and the target values ​​of the second quantization parameter, so as to obtain the quantized key-value matrix of the second format. The dequantization function is used to perform floating-point operations on the key-value matrix of the second format based on the target values ​​of the first quantization parameter and the second quantization parameter, to obtain the dequantized key-value matrix of the first format.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images