Quantization parameter storage method, model inference method, apparatus, electronic device, storage medium, and program

Abstract

The present disclosure provides a quantization parameter storage method and apparatus, a model inference method and apparatus, an electronic device, and a storage medium.SOLUTION: Statistically obtaining, by a calculating unit of a processor, a statistical value of a first quantization parameter of a model according to reference data, searching, by the calculating unit, a search space for a target value of the first quantization parameter and a target value of a second quantization parameter of the model according to the statistical value of the first quantization parameter, and saving, by the calculating unit, the target value of the first quantization parameter and the target value of the second quantization parameter to a memory.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to the fields of computer technology, and more particularly to the fields of large-scale model technology, artificial intelligence technology, and model quantization technology. [Background technology]

[0002] The cost of model inference increases significantly with the number of model parameters and the amount of context. Large models have a huge number of parameters and context. For example, some models have tens of billions of parameters, and some have millions of words of context. Low-bit quantization reduces the video memory footprint of graphics processing units (GPUs), thereby reducing the deployment costs of large models. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides a method for storing quantization parameters, a model inference method, an apparatus, an electronic device, a storage medium, and a program. [Means for solving the problem]

[0004] In one aspect of the present disclosure, there is provided a method for storing a quantization parameter, the method comprising: Statistically obtaining, by a calculation unit of the processor, a statistical value of the first quantization parameter of the model based on the reference data; The calculation unit searches a search space to obtain a target value of the first quantization parameter and a target value of a second quantization parameter of a model according to the statistical value of the first quantization parameter; and storing, by the calculation unit, the target value of the first quantization parameter and the target value of the second quantization parameter in a memory.

[0005] In another aspect of the present disclosure, there is provided a model inference method, the method comprising: processing the model input data with a computational unit of the processor to obtain a key-value matrix in a first format; reading, by the computing unit, a target value of a first quantization parameter and a target value of a second quantization parameter from a memory, the target value of the first quantization parameter and the target value of the second quantization parameter being stored in the memory prior to a model inference process using any of the methods described above; quantizing the key value matrix of the first format according to a quantization function established by the target value of the first quantization parameter and the target value of the second quantization parameter by the calculation unit to obtain a key value matrix of a second format; storing, by the computing unit, the key-value matrix in the second format in a key-value cache of the processor.

[0006] In another aspect of the present disclosure, there is provided a storage device for quantization parameters, the device comprising: a statistical module for statistically obtaining, by a calculation unit of the processor, a statistical value of the first quantization parameter of the model based on the reference data; a search module for searching a search space to obtain a target value of the first quantization parameter and a target value of a second quantization parameter of a model according to the statistical value of the first quantization parameter by the calculation unit; The calculation unit includes a calculation module for storing the target value of the first quantization parameter and the target value of the second quantization parameter in a memory.

[0007] In another aspect of the present disclosure, there is provided a model inference apparatus, the apparatus comprising: a processing module for processing the input data of the model with a computational unit of the processor to obtain a key-value matrix in a first format; a first reading module for reading, by the calculation unit, a target value of the first quantization parameter and a target value of the second quantization parameter from a memory, the target value of the first quantization parameter and the target value of the second quantization parameter being stored in the memory before a model inference process using any of the above-mentioned devices; a quantization module for quantizing the first-format key value matrix according to a quantization function established by the calculation unit using the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a second-format key value matrix; and a storage module for storing, by the computing unit, the key-value matrix in the second format in a key-value cache of the processor.

[0008] In another aspect of the present disclosure, there is provided an electronic device, the device comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the implementation of any one of the methods in the embodiments of the present disclosure.

[0009] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium having stored thereon computer instructions for causing a computer to perform any one of the methods in the embodiments of the present disclosure.

[0010] Another aspect of the present disclosure provides a program that, when executed by a processor, performs any of the methods of the embodiments of the present disclosure.

[0011] According to the present disclosure, quantization parameters are pre-calculated and stored, and then in the inference process, the pre-calculated quantization parameters can be read out based on specific usage needs to perform quantization inference, thereby reducing the processor memory occupation amount.Since the quantization parameters do not need to be repeatedly calculated in the inference process, the computational resources required for the inference process can be reduced, and the inference speed and efficiency can be improved.

[0012] It should be understood that the contents described herein are not intended to describe key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be better understood through the following specification.

[0013] The accompanying drawings are for better understanding of the solutions of the present disclosure and are not to be construed as limiting the present disclosure. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of a large-scale language model according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram illustrating a configuration of a multi-head attention layer according to an embodiment of the present disclosure. [Figure 3] 1 is a flowchart of a method for storing quantization parameters according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. [Figure 5] 10 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. [Figure 6] 10 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. [Figure 7] 10 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. [Figure 8]1 is a flowchart of a model inference method according to one embodiment of the present disclosure. [Figure 9] 10 is a flowchart of a model inference method according to another embodiment of the present disclosure. [Figure 10] FIG. 1 is a schematic diagram illustrating an application scenario of ABQ according to one embodiment of the present disclosure. [Figure 11] FIG. 1 is a schematic diagram illustrating a configuration of a storage device for quantization parameters according to an embodiment of the present disclosure. [Figure 12] FIG. 10 is a schematic diagram illustrating a configuration of a storage device for quantization parameters according to another embodiment of the present disclosure. [Figure 13] FIG. 1 is a schematic diagram illustrating a configuration of a model inference device according to an embodiment of the present disclosure. [Figure 14] FIG. 10 is a schematic diagram illustrating a configuration of a model inference device according to another embodiment of the present disclosure. [Figure 15] FIG. 1 is a block diagram of an electronic device for implementing the method of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015]

[0023] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as illustrative only. Therefore, it should be understood that those skilled in the art can make various changes and modifications to the embodiments described herein without departing from the scope of the present disclosure. Similarly, descriptions of well-known features and structures are omitted from the following description for clarity and conciseness.

[0016] Large-scale models are being applied globally. For example, there are large-scale models that solve problems such as dialogue, logical thinking, code generation, and knowledge question-and-answering. They are also being applied and implemented in various scenarios in the Chinese language, including large-scale models with up to 70 billion parameters and models generated with 2 million words of context. For example, a model with 70 billion parameters requires 140GB of GPU video memory space for inference, and when combined with 2 million words of context (approximately 400GB), more than 500GB of video memory is required. Considering that some GPUs have a single card with 80GB of video memory, without optimization, eight cards would be required, which would only support one user at a time and would result in high inference costs.

[0017] Low-bit key-value cache (KV cache) (hereafter abbreviated as C4 and C2) quantization includes dynamic quantization and mixed quantization. However, these methods have several issues. For example, the dynamic C4 quantization method requires the calculation of information such as the quantization scaling factor (scale) for each query during the decoding process to ensure quantization accuracy. The need to repeatedly calculate information such as the quantization scale during the inference process results in inference overhead and does not meet practical implementation needs. Furthermore, quantizing mixed bits in C4 and C8 requires operations such as modifying the model network to ensure inference effectiveness. Another example is that non-quantized inference methods can also consume additional time during inference. For example, prompt compression requires the compression of a small-scale model prior to inference, which requires inference time. Token elimination requires a specific elimination strategy and calculations in conjunction with the token.

[0018] FIG. 1 is a schematic diagram illustrating the model structure. A solution based on an embodiment of the present disclosure can propose a low-cost inference deployment method for large language models (LLMs). LLMs are used to solve common natural language tasks such as semantic understanding, multi-round dialogue chat, logical reasoning, code creation, and writing. The model structure consists of a stack of several transformer layers, each of which contains models such as layer normalization (LayerNorm), multi-head attention, and a fully connected layer (FeedForward). Input text is processed through a text and position embedding representation layer to obtain features such as text vectors and text tensors.

[0019] To reduce the cost of inference (post-training), the above model structure requires the multi-head attention module to store key-value cache (KV cache) information. When the model input information is long (e.g., a scenario involving 2 million words), both the video memory and computational complexity required are large, resulting in high inference costs for large-scale models. As shown in Figure 2, the multi-head attention module architecture requires the storage of key (K) and value (V) values ​​during the actual inference process, which requires repeated storage and retrieval, consuming large computational bandwidth and GPU video memory. For example, the scaled dot-product attention module can perform matrix multiplication (MatMul), scaling, masking, and softmax operations on the query (Q) matrix and key (K), and then perform matrix multiplication (MatMul) on the calculated result and the value (V) matrix.

[0020] 3 is a flowchart of a method for storing quantization parameters according to an embodiment of the present disclosure.

[0021] In S310, the calculation unit of the processor statistically obtains the statistical value of the first quantization parameter of the model based on the reference data.

[0022] In S320, the calculation unit searches a search space to obtain a target value of the first quantization parameter and a target value of a model second quantization parameter according to the statistical value of the first quantization parameter.

[0023] In S330, the calculation unit stores the target value of the first quantization parameter and the target value of the second quantization parameter in a memory.

[0024] In an embodiment of the present disclosure, the processor may be a graphics processing unit (GPU), a central processing unit (CPU), or a neural processing unit (NPU) (or neural network processing unit), etc. The processor may include a computation unit, a memory, a cache, etc. For example, the computation unit in a GPU may include a stream multiprocessor (SM), and the memory may also be referred to as a video memory.

[0025] In an embodiment of the present disclosure, quantization can be used to reduce processor cache occupation during model inference for a trained model. Various quantization parameters can be used during model inference, such as a quantization scaling factor (scale) and a quantization zero point (zero_point). If a dynamic quantization scheme is adopted, the quantization parameters must be repeatedly calculated during the inference process, consuming significant computational resources. An embodiment of the present disclosure can also adopt a static quantization scheme, in which the quantization parameters required for inference are calculated in advance and stored in memory, such as a hard disk.

[0026] In an embodiment of the present disclosure, statistical values ​​of the quantization parameters of a model can be calculated based on reference data. The reference data can be extracted from training samples of the model. For example, the reference data can include text information. Referring to FIG. 1, the reference data can be input into the model and then embedded coded to obtain reference features, such as reference vectors and reference tensors.

[0027] In an embodiment of the present disclosure, the model may have an attention layer, such as a self-attention layer or a multi-head self-attention layer. A statistical rule may be set in the attention layer. Referring to FIG. 1, the calculation unit may perform processing such as a normalization layer on the reference features before inputting them into the attention layer. The attention layer may perform statistical calculations on the received features based on the statistical rule to obtain statistical values ​​for the quantization parameters. Statistical rules include statistical mean values, maximum of statistical absolute maximum values, etc. Based on the mean value statistical rule, the average value of multiple reference data may be calculated, and based on the maximum of absolute maximum values ​​statistical rule, the maximum of absolute maximum values ​​of multiple reference data may be calculated. Then, based on the statistical results, the statistical values ​​for the first quantization parameter of the model may be calculated. The statistical values ​​for the first quantization parameter may be stored in a cache or memory. Furthermore, a pre-stored search space may be read from memory, and target values ​​for the first quantization parameter and the second quantization parameter of the model may be obtained based on the search value in the search space and the statistical value of the first quantization parameter. In an embodiment of the present disclosure, the first quantization parameter may be a quantization scaling factor (scale), and the second quantization parameter may be a quantization zero point (zero_point). The calculation unit may store the target value of the first quantization parameter and the target value of the second quantization parameter in a memory such as a hard disk.

[0028] In the embodiments of the present disclosure, the quantization parameters can be pre-calculated and stored, and in the inference process, the pre-calculated quantization parameters can be read out to perform quantization inference according to specific usage needs, thereby reducing the memory occupation of the processor.Since the quantization parameters do not need to be repeatedly calculated in the inference process, the computation resources occupied in the inference process can be reduced, and the inference speed and efficiency can be improved.

[0029] In one embodiment, the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a model inference process and used to quantize a first-format key-value matrix required for an attention layer of a model into a second-format key-value matrix, where the second-format key-value matrix is ​​stored in a key-value cache of the processor.

[0030] In an embodiment of the present disclosure, during the model inference process, the processor can retrieve required quantization parameters from memory based on the quantization needs of each layer of the model. For example, if quantization is required for the key-value (KV) matrix of the attention layer of the model, a first quantization parameter, such as a target value of a quantization scaling factor, and a second quantization parameter, such as a target value of a quantization zero point, can be retrieved from memory to quantize the key-value matrix required for the attention layer. For example, before quantization, the first format of the key-value matrix required for the attention layer is a 16-bit floating-point (Brain Floating Point 16, BF16) format. After quantization, the second format of the key-value matrix required for the attention layer is a 4-bit integer (INT4) format. The INT4 format occupies less storage space than the BF16 format. The key-value matrix required for the attention layer can be stored in the processor's key-value cache (KV cache) after quantization. The quantized KV matrix occupies less storage space than the unquantized KV matrix, and since the key-value cache is usually located in the processor's memory, the amount of processor memory, such as the GPU's video memory, required can be reduced. Quantization can be performed using a pre-calculated and stored quantization parameter KV matrix, eliminating the need to first calculate the quantization parameters before quantization. This reduces the computing resources required for the inference quantization process and improves inference speed and efficiency.

[0031] In one embodiment, the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a model inference process, and used to dequantize the second-format key-value matrix read from the key-value cache into a first-format key-value matrix, where the dequantized key-value matrix is ​​used as an input feature of the attention layer.

[0032] For example, when a second-format key-value (KV) matrix in a key-value cache (KV cache) needs to be dequantized, the processor can read a first quantization parameter, e.g., a target value of a quantization scaling coefficient, and a second quantization parameter, e.g., a target value of a quantization zero point, from memory and dequantize them for the key-value matrix required for the attention layer. The first quantization parameter may be a dequantization scaling coefficient, or the dequantization scaling coefficient may be estimated from the quantization scaling coefficient. For example, dequantization can dequantize a second-format key-value matrix into a first-format key-value matrix. After dequantization, the accuracy of the key-value matrix input to the attention layer of the model can be improved. Since the quantization parameter can be calculated and stored in advance, dequantization can be performed on the quantized KV matrix in memory, eliminating the need to first calculate the quantization parameter and then dequantize it. This reduces the computational resources required for the dequantization process, thereby improving inference speed and efficiency.

[0033] 4 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features of the above-described embodiments. In one embodiment, statistically obtaining, by a calculation unit of a processor, a statistical value of the first quantization parameter of the model based on reference data includes at least one of the following:

[0034] In S410, the average minimum value and the average maximum value of the first quantization parameter are statistically obtained by averaging the reference data.

[0035] In S420, the minimum value of the absolute maximum values ​​and the maximum value of the absolute maximum values ​​of the first quantization parameter are statistically obtained according to the absolute maximum value for the reference data.

[0036] In the embodiments of the present disclosure, there are no limitations on the timing of S410 and S420, and S410 may be executed first and then S420, or S420 may be executed first and then S410, or only one of the steps may be executed.

[0037] For example, when one reference data corresponds to a set of features, the average value and the maximum absolute maximum value of the set of features can be calculated first. The average value statistical rule can be used to statistically obtain the average minimum value and the average maximum value from the average values ​​corresponding to N reference data. The maximum absolute maximum value statistical rule can be used to statistically obtain the minimum absolute maximum value and the maximum absolute maximum value from the absolute maximum values ​​corresponding to N reference data. The target value of the first quantization parameter can be more accurately searched based on one or more of the average minimum value, the minimum absolute maximum value, the average maximum value, and the maximum absolute maximum value.

[0038] 5 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features of the above-described embodiments. In one embodiment, searching a search space to obtain a target value of the first quantization parameter and a target value of a second quantization parameter of a model based on statistics of the first quantization parameter includes:

[0039] At S510, candidate values ​​of the first quantization parameter and candidate values ​​of the second quantization parameter are calculated based on the average minimum value, the minimum absolute maximum value, the average maximum value, the maximum absolute maximum value and search parameters in the search space.

[0040] In S520, the loss function values ​​are calculated based on the corresponding candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter for all search parameters in the search space and the key value matrix of the reference data, and the target search parameters that minimize the value of the loss function are searched for and obtained.

[0041] At S530, a target value of the first quantization parameter and a target value of the second quantization parameter are calculated based on the target search parameters.

[0042] In an embodiment of the present disclosure, the search parameters can include multiple search values. One or more search spaces can be pre-set in the processor's memory. The calculation unit can read the search values ​​of the search spaces from the memory and use them in subsequent calculations. For example, the search space S = [0, 0.2, 0.4, 0.6]. The calculation unit can use similar search parameters when calculating candidate values ​​for the first quantization parameter and the second quantization parameter. For example, a search parameter s such as 0.2 can be selected from the search space S and substituted into the related equation for the quantization parameters to calculate candidate values ​​for the first quantization parameter and the second quantization parameter. Next, the calculated candidate values ​​for the first quantization parameter and the second quantization parameter are substituted into the loss function equation to calculate the corresponding loss value of s. The corresponding loss values ​​of all s in the search space are compared, and the s with the smallest loss value is selected. o Then, the target search parameter s o The target value of the first quantization parameter and the target value of the second quantization parameter can be calculated by substituting the above into the relational expression of the quantization parameters.

[0043] In the embodiments of the present disclosure, the target value of the quantization parameter can be quickly searched for based on the search space, thereby improving calculation speed and efficiency. Furthermore, the search parameters in the search space can be optimized according to the search process. For example, if the loss value corresponding to a search parameter is greater than a threshold, the search parameter can be deleted. Furthermore, if the larger the search parameter, the larger the corresponding loss value, the more search parameters with smaller values ​​can be selected. Furthermore, if the larger the search parameter, the smaller the corresponding loss value, the more search parameters with larger values ​​can be selected. Optimizing the search space can further improve search speed and efficiency.

[0044] 6 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features of the above-described embodiments. In one embodiment, calculating candidate values ​​for the first quantization parameter and the second quantization parameter based on an average minimum value, a minimum of absolute maximum values, an average maximum value, a maximum of absolute maximum values, and search parameters in the search space includes:

[0045] At S610, a minimum value of the first quantization parameter is calculated based on the average minimum value, the minimum value of the absolute maximum values, and search parameters in the search space.

[0046] At S620, the maximum value of the first quantization parameter is calculated based on the average maximum value, the maximum value of the absolute maximum value, and search parameters in the search space.

[0047] At S630, candidate values ​​for the first quantization parameter are calculated based on the maximum and minimum values ​​of the first quantization parameter.

[0048] At S640, the candidate value of the second quantization parameter is calculated based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter.

[0049] An example of the formula for the minimum value of one first quantization parameter is as follows:

number

[0050] An example of the formula for the maximum value of one first quantization parameter is as follows:

number

[0051] An example of the formula for one candidate value of the first quantization parameter is as follows:

number

[0052] An example of the formula for one candidate value of the second quantization parameter is as follows:

number

[0053] For example, the minimum quantization scaling coefficient value scale_min can be calculated by substituting the statistically obtained average minimum value avg_min, the minimum absolute maximum value absmax_min, and any search parameter s selected from the search space into Equation 1 above. The maximum quantization scaling coefficient value scale_max can be calculated by substituting the statistically obtained average maximum value avg_max, the maximum absolute maximum value absmax_max, and any search parameter s selected from the search space into Equation 2 above. Substituting scale_min and scale_max into Equation 3 for candidate values ​​of the first quantization parameter gives a candidate value scale for the first quantization parameter. Next, substituting scale_min and scale into Equation 4 for candidate values ​​of the second quantization parameter gives a candidate value zero_point for the second quantization parameter.

[0054] In the embodiment of the present disclosure, the candidate quantization parameter values ​​calculated based on multiple statistical values ​​of the quantization parameters are more accurate, and the same search parameters in the same search space can be used to improve the search speed.

[0055] 7 is a flowchart of a method for storing quantization parameters according to another embodiment of the present disclosure. This embodiment may include one or more features of the above-described embodiments. In one embodiment, calculating candidate values ​​for the first quantization parameter and the second quantization parameter based on an average minimum value, a minimum of absolute maximum values, an average maximum value, a maximum of absolute maximum values, and search parameters in the search space includes:

[0056] At S710, a minimum value of the first quantization parameter is calculated based on the average minimum value, the minimum of the absolute maximum values, and a first search parameter in a first search space.

[0057] At S720, the maximum value of the first quantization parameter is calculated based on the average maximum value, the maximum of the absolute maximum values, and a second search parameter in a second search space.

[0058] At S730, candidate values ​​of the first quantization parameter are calculated based on the maximum and minimum values ​​of the first quantization parameter.

[0059] At S740, the candidate value of the second quantization parameter is calculated based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter.

[0060] An example of the formula for the minimum value of one first quantization parameter is as follows:

number

[0061] An example of the formula for the maximum value of one first quantization parameter is as follows:

number

[0062] In the embodiments of the present disclosure, s1 and s2 may be different search parameters in different search spaces. In one embodiment, s1 and s2 may be different search parameters in the same search space, in which case the first search space and the second search space can be understood as the same search space.

[0063] After calculating scale_min and scale_max based on Equations 5 and 6, scale and zero_point can be calculated by referring to Equations 4 and 5.

[0064] For example, the minimum value of the quantization scaling coefficient, scale_min, can be calculated by substituting the statistically obtained average minimum value, avg_min, the minimum value of the absolute maximum value, absmax_min, and any first search parameter s1 selected in the first search space S1 into the above equation 5. The maximum value of the quantization scaling coefficient, scale_max, can be calculated by substituting the statistically obtained average maximum value, avg_max, the maximum value of the absolute maximum value, absmax_max, and any second search parameter s2 selected in the second search space S2 into the above equation 6. The candidate value of the first quantization parameter, scale, can be obtained by substituting scale_min and scale_max into equation 3 for the candidate value of the first quantization parameter. Next, the candidate value of the second quantization parameter, zero_point, can be obtained by substituting scale_min and scale into equation 4 for the candidate value of the second quantization parameter.

[0065] In the embodiment of the present disclosure, the candidate quantization parameter values ​​calculated based on multiple statistical values ​​of the quantization parameters are more accurate. Using different search parameters can improve the accuracy of the search results.

[0066] In one embodiment, the loss function is determined based on a quantization function and an inverse quantization function.

[0067] In one embodiment, the quantization function is used to perform a rounding operation on the key value matrix of the first format of the reference data based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix of the second format after quantization.

[0068] An example of one quantization function is:

number

[0069] In one embodiment, the inverse quantization function is used to perform floating-point operations on the quantized second-format key value matrix based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the inverse-quantized first-format key value matrix.

[0070] An example of one inverse quantization function is:

number

[0071] An example formula for one loss function is:

number

[0072] For example, by substituting the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter obtained by calculation using the above Equations 3 and 4, and the current reference data x, into Equation 7, the key value matrix quant_x of the current reference data x in the second format can be quantized. By substituting the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization factor obtained by calculation using the above Equations 3 and 4, and the current key value matrix quant_x of the second format into Equation 8, the current reference data dequant_quant_x of the first format can be quantized. Substituting the above quant_x and dequant_quant_x into the loss function Equation 9, the loss value under the current search parameters can be obtained. An optimal configuration s that minimizes the loss value can then be adaptively searched for and obtained, and the search parameters corresponding to the minimum loss value and the corresponding target values ​​of the first quantization parameter and the second quantization parameter can be obtained.

[0073] According to an embodiment of the present disclosure, a loss function can be constructed based on the multiplication result of the quantization function and the inverse quantization function. The resulting loss function is more suited to the actual situation of information loss in the quantization and inverse quantization of reference data using the quantization scaling coefficient and the quantization zero point under the current search parameters. The loss function can be used to obtain the optimal quantization scaling coefficient and the quantization zero point for the quantization and inverse quantization of the current reference data. This can improve the accuracy of model inference and reduce the memory (such as video memory) occupied by model inference.

[0074] 8 is a flowchart of a model inference method according to one embodiment of the present disclosure. The method includes:

[0075] At S810, the input data of the model is processed by a computation unit of the processor to obtain a key-value matrix in a first format.

[0076] At S820, the calculation unit reads the target value of the first quantization parameter and the target value of the second quantization parameter from the memory, where the target value of the first quantization parameter and the target value of the second quantization parameter are stored in the memory before the model inference process using any of the above-mentioned quantization parameter storage methods.

[0077] At S830, the calculation unit quantizes the first format key value matrix based on a quantization function constructed by the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a second format key value matrix.

[0078] At S840, the calculation unit stores the second format key-value matrix in a key-value cache of the processor.

[0079] In an embodiment of the present disclosure, the processor may be a graphics processing unit (GPU), a central processing unit (CPU), or a neural processing unit (NPU) (or neural network processing unit), etc. The processor may include a calculation unit, a memory, a cache, etc. For example, the calculation unit in a GPU may include a stream multiprocessor (SM), and the memory may also be referred to as video memory. The memory may be a bit hard disk such as a solid state disk (SSD) or a mechanical hard disk drive (HDD).

[0080] In an embodiment of the present disclosure, a processor extracts features from model input data, such as text, to obtain feature data. The feature data is in a first format, such as a BF16 key-value matrix. Based on the current BF16 feature data, corresponding target values ​​of a first quantization parameter and a second quantization parameter can be read from memory. The target values ​​of the first quantization parameter and the second quantization parameter may be quantization parameters that are obtained by calculating reference data using the above-described quantization parameter storage method before model inference (or during the model deployment stage) and stored in memory. Here, the reference data may be data selected from sample data.

[0081] In an embodiment of the present disclosure, a key-value cache (KV cache) can be stored in the memory of the processor, and a key-value matrix in a first format can be quantized based on the read target values ​​of the first quantization parameter and the second quantization parameter to obtain a key-value matrix in a second format, for example, int4, whose data amount is smaller than that of the key-value matrix in the first format. The key-value matrix in the second format obtained by quantization can be stored in a key-value cache in the memory (video memory) of the processor, for example, a GPU.

[0082] According to an embodiment of the present disclosure, when a reference model to be processed by a model is quantized into a key-value matrix in a second format, such as int4, which has a smaller amount of data, and stored in the memory of a computing unit, the quantization takes up less storage space than a key-value matrix in a first format, such as BF16, and memory resource consumption can be reduced. With the same memory resources, more users can be supported. Compared to reading statically stored, pre-obtained quantization parameters from memory, performing quantization, and calculating the quantization parameters, the efficiency and applicability of quantization can be improved.

[0083] 9 is a flowchart of a model inference method according to another embodiment of the present disclosure. This embodiment may include one or more features of the above-described embodiments. In one embodiment, the method further includes:

[0084] In S910, the calculation unit reads the second format key-value matrix from the key-value cache.

[0085] At S920, the calculation unit inverse-quantizes the second-format key value matrix based on an inverse quantization function constructed by the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a first-format key value matrix, which is input into the attention layer of the model.

[0086] In the embodiment of the present disclosure, when the attention layer of the model needs to input data for inference, it first extracts the key-value matrix in the second format from the key-value cache in the memory, and then performs inverse quantization using the corresponding target values ​​of the first quantization parameter and the second quantization parameter to obtain the key-value matrix in the first format, and then inputs the key-value matrix in the first format into the attention layer of the model for inference.

[0087] According to an embodiment of the present disclosure, a key-value matrix in a second format in a key-value cache can be extracted and dequantized to a key-value matrix in a first format before quantization, thereby reducing the memory occupation amount of the key-value cache and improving the accuracy of model inference.

[0088] In one embodiment, the quantization function is used to perform a rounding operation on a key value matrix in a first format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in a second format after quantization.

[0089] In one embodiment, the inverse quantization function is used to perform floating-point operations on the second-format key value matrix based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the first-format key value matrix after inverse quantization.

[0090] In an embodiment of the present disclosure, the data in the first format is data required for model inference, input to a model attention layer, and stored in the memory of a computing unit. The data in the second format is obtained based on the data in the first format and occupies less memory resources than the data in the first format.

[0091] In an embodiment of the present disclosure, when it is necessary to use the memory of a computing unit to store the key value matrix, the model in the first format can be quantized into data in the second format based on a quantization method. The basis for quantization may be a target value of a first quantization parameter and a target value of the second quantization parameter. When it is necessary to perform inference using the model, the data in the second format can be dequantized into data in the first format, and the basis for dequantization may be the target value of the first quantization parameter and the target value of the second quantization parameter used when quantizing the data in the first format.

[0092] According to an embodiment of the present disclosure, data in a second format, which occupies less memory resources, can be stored in the memory of a computing unit, and when performing model inference, the data in the second format can be read from the memory, dequantized to data in the first format, and input to the attention layer of the model, thereby reducing the memory occupation amount of the key-value cache and improving the accuracy of model inference.

[0093] The model inference method of the present disclosure includes a static 4-bit (4-bit) key-value cache (KV cache) quantization scheme for large-scale model inference, which reduces video memory consumption during inference and improves inference speed. For example, by reducing the KV cache from 16 bits to 4 bits, the video memory overhead of 400 GB can be reduced to 100 GB. With the same GPU resource consumption, the model can be optimized from supporting one user to supporting four users simultaneously with the same number of resources, thereby increasing the query rate per second (QPS) by three times. Applying this scheme to large-scale model inference optimization scenarios not only improves the efficiency of large-scale model usage to near-lossless, but also reduces inference memory overhead, improves QPS, and reduces the inference deployment cost of large-scale models.

[0094] In one application scenario, K and V in a model can be stored as 4-bit integers (INT4) instead of 16-bit floating-point numbers (Brain Floating Point16, BF16). This accelerates the access process for K and V, reduces the video memory required for inference (by 75%), significantly improves QPS for large models, and reduces inference deployment costs. To achieve efficient and accurate INT4 storage targets, an adaptive bagging quantization (ABQ) algorithm can be employed. For example, this can include two rounds of calibration.

[0095] In the first round, the calibration data (which may be extracted from training samples) is statistically analyzed using the average value (avg) and absolute maximization (absmax) quantization methods, and the minimum value (scale_min) and maximum value (scale_max) of the quantization scaling coefficients are calculated.

[0096] In the second round, we search the custom search space for the quantization scaling factors (scale) of K and V in each layer that minimizes the value of the loss function (Loss), and calculate the corresponding quantization zero point (zero_point).

[0097] In one embodiment, scale_min and scale_max in a single calculation can share the same s. In another embodiment, scale_min and scale_max in a single calculation can use different s.

[0098] In the case of multiple data, you can calculate one or more of the mean, minimum, and maximum values ​​for the loss under the same K or V for different samples, and then select the scale and zero_point that have the best effect.

[0099] The embodiments of the present disclosure can be widely applied to large-scale model inference deployment, such as general-purpose model acceleration and long-text model deployment, to reduce inference costs and significantly improve QPS. In one application scenario, as shown in Figure 10, the up-down arrows represent data streams, and the left-right arrows represent model streams. During model training, the model receives training data and generates an original model with a BF16 (bfloat16) data type, which has a high inference deployment cost. After the ABQ process, the model's KVcache data type is converted from BF16 to INT4. During inference, the C4 model generated by ABQ can be repeatedly invoked to provide feedback on the answer to the user query.

[0100] The static C4 quantization scheme (including the ABQ process) of the embodiments of the present disclosure performs C4 quantization without introducing additional inference overhead (e.g., time-consuming), reduces video memory usage, improves QPS, and reduces inference costs, for example, to 25% of the original cost. For large-scale model products, the C4 quantization scheme is an important acceleration method, which can reduce costs by more than two times without affecting the user experience. Given the large number of users of large-scale models, this scheme has significant potential for cost reduction, thereby effectively supporting low-cost applications of large-scale models.

[0101] 11 is a schematic diagram illustrating the configuration of a storage device for quantization parameters according to an embodiment of the present disclosure. In one embodiment, the device includes:

[0102] a statistics module 1110 for statistically obtaining, by a calculation unit of the processor, a statistical value of the first quantization parameter of the model based on reference data;

[0103] A search module 1120 for searching a search space to obtain a target value of the first quantization parameter and a target value of a second quantization parameter of a model according to the statistical value of the first quantization parameter by the calculation unit;

[0104] The calculation unit includes a calculation module 1130 for storing the target value of the first quantization parameter and the target value of the second quantization parameter in a memory.

[0105] In one embodiment, the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a model inference process and used to quantize a first-format key-value matrix required for an attention layer of a model into a second-format key-value matrix, where the second-format key-value matrix is ​​stored in a key-value cache of the processor.

[0106] In one embodiment, the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a model inference process, and used to dequantize the second-format key-value matrix read from the key-value cache into a first-format key-value matrix, where the dequantized key-value matrix is ​​used as an input feature of the attention layer.

[0107] In one embodiment, the statistics module 1110: Statistically obtaining an average minimum value and an average maximum value of the first quantization parameter by averaging the reference data; and statistically obtaining a minimum value of the absolute maximum values ​​and a maximum value of the absolute maximum values ​​of the first quantization parameter using the absolute maximum value for the reference data.

[0108] 12 is a schematic diagram illustrating the configuration of a quantization parameter storage device according to another embodiment of the present disclosure. The device may include one or more features of the quantization parameter storage device described above. In one embodiment, the search module 1120: a first calculation sub-module 1121 for calculating candidate values ​​of the first quantization parameter and the second quantization parameter according to an average minimum value, a minimum absolute maximum value, an average maximum value, a maximum absolute maximum value and search parameters in the search space; a search submodule 1122 for calculating a loss function value based on the corresponding candidate values ​​of the first quantization parameter and the second quantization parameter of all search parameters in a search space and the key value matrix of the reference data, and searching for and obtaining a target search parameter that minimizes the value of the loss function; a second calculation sub-module 1123 for calculating a target value of the first quantization parameter and a target value of the second quantization parameter according to the target search parameter.

[0109] In one embodiment, the first calculation sub-module 1121 is: calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a search parameter in the search space; calculating a maximum value of the first quantization parameter based on the average maximum value, the maximum of the absolute maximum values, and a search parameter in the search space; calculating candidate values ​​for the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; and calculating the candidate value of the second quantization parameter based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter.

[0110] In one embodiment, the first calculation sub-module 1121 is: calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a first search parameter in a first search space; calculating a maximum value of the first quantization parameter based on the average maximum, the maximum of the absolute maximum values, and a second search parameter in a second search space; calculating candidate values ​​for the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; and calculating the candidate value of the second quantization parameter based on the minimum value of the first quantization parameter and the candidate value of the first quantization parameter.

[0111] In one embodiment, the loss function is determined based on a quantization function and an inverse quantization function; the quantization function is used to perform a rounding operation on the key value matrix of the first format of the reference data based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix of the second format after quantization; The inverse quantization function is used to perform floating-point operations on the quantized second-format key value matrix based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the inverse-quantized first-format key value matrix.

[0112] 13 is a schematic diagram illustrating the configuration of a model inference device according to an embodiment of the present disclosure. In one embodiment, the device includes: a processing module 1310 for processing the input data of the model by a computation unit of the processor to obtain a key-value matrix in a first format; a first reading module 1320 for reading, by the computing unit, a target value of a first quantization parameter and a target value of a second quantization parameter from a memory, the target value of the first quantization parameter and the target value of the second quantization parameter being stored in the memory before the model inference process using any of the quantization parameter storage devices described above; a quantization module 1330 for quantizing the first-format key value matrix according to a quantization function established by the calculation unit using the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a second-format key value matrix; and a storage module 1340 for storing, by the computation unit, the key-value matrix in the second format in a key-value cache of the processor.

[0113] 14 is a schematic diagram illustrating the configuration of a model inference apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the model inference apparatus described above. In one embodiment, the apparatus includes: a second reading module 1410 for reading, by the computing unit, the key-value matrix in the second format from the key-value cache; The apparatus further includes an inverse quantization module 1420 for inverse quantizing the second-format key value matrix based on an inverse quantization function constructed by the calculation unit using the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a first-format key value matrix and input it to the attention layer of the model.

[0114] In one embodiment, the quantization function is used to perform a rounding operation on a key value matrix in a first format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in a second format after quantization; The inverse quantization function is used to perform floating-point operations on the key value matrix in the second format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in the first format after inverse quantization.

[0115] For specific functions and exemplary descriptions of each module and sub-module of the apparatus according to the embodiments of the present disclosure, please refer to the relevant descriptions of the corresponding steps in the above-mentioned method embodiments, and they will not be repeated here.

[0116] In the technical solution of the present disclosure, the acquisition, storage, and application of users' personal information comply with the provisions of relevant laws and regulations and do not violate public order and morals.

[0117] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a non-transitory computer-readable storage medium, and a program product.

[0118] 15 is a block diagram of an electronic device 1500 for implementing an embodiment of the present disclosure. The electronic device refers to various types of digital computers, including laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device also refers to various types of mobile devices, including personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computing devices. The components, their interconnections, and functions described in this disclosure are merely exemplary and do not limit the implementation of what is described and specified in this disclosure.

[0119] 15, device 1500 includes a computing unit 1501 that can perform various appropriate operations and processes based on computer program instructions stored in a read-only memory (ROM) 1502 or loaded from a storage unit 1508 into a random access memory (RAM) 1503. The RAM 1503 can further store various programs and data required for the operation of device 1500. The computing unit 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0120] The components of device 1500 are connected to an I / O interface 1505, which includes an input unit 1506 such as a keyboard or mouse, an output unit 1507 such as various displays and speakers, a storage unit 1508 such as a magnetic disk or optical disk, and a communication unit 1509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 1509 allows device 1500 to exchange information / data with other devices via a computer network such as the Internet and / or various carrier networks.

[0121] The computing unit 1501 may be a variety of general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, computing units that execute various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1501 performs each of the methods and processes described above, such as the quantization parameter storage method and / or the model inference method. For example, in some embodiments, the quantization parameter storage method and / or the model inference method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1508. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 1500 via the ROM 1502 and / or the communication unit 1509. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, it can perform one or more steps of the quantization parameter storage method and / or the model inference method described above. Additionally, in other embodiments, the computing unit 1501 can be configured to perform the quantization parameter storage method and / or the model inference method in any other suitable manner (e.g., firmware).

[0122] Various embodiments of the systems or techniques described in this disclosure may be implemented using digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may involve execution by one or more computer programs executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, capable of receiving data and instructions from, and transferring data and instructions to, a storage system, at least one input device, and at least one output device.

[0123] Program code for carrying out the methods of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programming data processing apparatus, such that when the program code is executed by the processor or controller, it can perform the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on-site, partially on-site, partially on-site and partially on a remote site as a separate soft encapsulation, or entirely on a remote site or server.

[0124] In the context of this disclosure, a machine-readable medium may be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Further examples of machine-readable storage media include one or more hard-wired electrical connections, a portable computer disk cartridge, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any combination of the foregoing.

[0125] To provide for user interaction, the systems and techniques described herein can be implemented on a computer that includes a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, etc.) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball, etc.) for the user to provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, haptic feedback, etc.), and input from the user can be received in any form (e.g., acoustic input, voice input, tactile input, etc.).

[0126] The systems and techniques described herein can be implemented in a computing system that includes background components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a user computer having a graphical user interface or network browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background, middleware, or front-end components. Components of the system can be connected to each other via any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] The computer system may include a client and a server. Typically, the client and server are remote from each other and generally interact via a communication network. The client-server relationship is created by a computer program running on a corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating a blockchain.

[0128] It should be understood that steps can be newly ranked, added, or deleted using the various aspects of the flow shown above. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order. This disclosure is not limited thereto, as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0129] The above specific examples do not constitute limitations on the scope of protection of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions are possible depending on design considerations and other factors. Any modifications, equivalent replacements, improvements, etc. within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for storing a quantization parameter, comprising: Statistically obtaining, by a calculation unit of the processor, a statistical value of the first quantization parameter of the large-scale model based on the reference data; The calculation unit searches for search parameters that minimize the value of a loss function in a search space based on the statistics of the first quantization parameter to obtain target search parameters, and calculates target values ​​of the first quantization parameter and the second quantization parameter of the large-scale model corresponding to the target search parameters based on the target search parameters; storing, by the calculation unit, the target value of the first quantization parameter and the target value of the second quantization parameter in a memory; How quantization parameters are stored.

2. The target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a large-scale model inference process, and are used to quantize a first-format key-value matrix required for an attention layer of the large-scale model into a second-format key-value matrix; wherein the key-value matrix in the second format is stored in a key-value cache of the processor. The method for storing quantization parameters according to claim 1 .

3. the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a large scale model inference process, and used to dequantize the second format key value matrix read from the key value cache into a first format key value matrix; Here, the dequantized key value matrix is ​​used as the input feature of the attention layer. The method for storing quantization parameters according to claim 2 .

4. Statistically obtaining, by a calculation unit of a processor, a statistical value of a first quantization parameter of a large-scale model based on reference data, Statistically obtaining an average minimum value and an average maximum value of the first quantization parameter by averaging the reference data; and statistically obtaining a minimum value of the absolute maximum values ​​and a maximum value of the absolute maximum values ​​of the first quantization parameter by an absolute maximum value with respect to the reference data. The method for storing quantization parameters according to claim 1 .

5. According to the statistical value of the first quantization parameter, search parameters that minimize the value of a loss function in a search space are obtained as target search parameters, and according to the target search parameters, target values ​​of the first quantization parameter and the second quantization parameter of the large-scale model corresponding to the target search parameters are calculated and obtained; calculating candidate values ​​for the first quantization parameter and candidate values ​​for the second quantization parameter based on an average minimum value, a minimum of absolute maxima, an average maximum value, a maximum of absolute maxima and search parameters in the search space; Calculating the value of the loss function based on the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter corresponding to all search parameters in a search space and the key value matrix of the reference data, and searching for and obtaining the search parameters that minimize the value of the loss function as target search parameters; calculating, based on the target search parameters, target values ​​of the first quantization parameter and the second quantization parameter corresponding to the target search parameters; The method for storing quantization parameters according to claim 4.

6. calculating candidate values ​​of the first quantization parameter and the second quantization parameter based on an average minimum value, a minimum of absolute maximum values, an average maximum value, a maximum of absolute maximum values, and search parameters in the search space; calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a search parameter in the search space; calculating a maximum value of the first quantization parameter based on the average maximum, the maximum value of the absolute maximum, and a search parameter in the search space; calculating candidate values ​​of the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; calculating candidate values ​​of the second quantization parameter based on a minimum value of the first quantization parameter and candidate values ​​of the first quantization parameter; The method for storing quantization parameters according to claim 5 .

7. calculating candidate values ​​of the first quantization parameter and the second quantization parameter based on an average minimum value, a minimum of absolute maximum values, an average maximum value, a maximum of absolute maximum values, and search parameters in the search space; calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a first search parameter in a first search space; calculating a maximum value of the first quantization parameter based on the average maximum, the maximum of the absolute maximum values, and a second search parameter in a second search space; calculating candidate values ​​of the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; calculating candidate values ​​of the second quantization parameter based on a minimum value of the first quantization parameter and candidate values ​​of the first quantization parameter; The method for storing quantization parameters according to claim 5 .

8. the loss function is determined based on a quantization function and an inverse quantization function; the quantization function is used to perform a rounding operation on a key value matrix of a first format of the reference data based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix of a second format after quantization; the inverse quantization function is used to perform floating-point arithmetic on the quantized key value matrix in the second format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the inverse quantized key value matrix in the first format. The method for storing quantization parameters according to claim 5 .

9. A large-scale model inference method, comprising: processing the input data of the large scale model with a computational unit of the processor to obtain a key-value matrix in a first format; reading, by the computing unit, a target value of a first quantization parameter and a target value of a second quantization parameter from a memory, the target value of the first quantization parameter and the target value of the second quantization parameter being stored in the memory prior to a large scale model inference process using the method of claim 1; quantizing the key value matrix in the first format according to a quantization function established by the target value of the first quantization parameter and the target value of the second quantization parameter by the calculation unit to obtain a key value matrix in a second format; storing, by the computing unit, the key-value matrix in the second format in a key-value cache of the processor; Large-scale model inference methods.

10. The large-scale model inference method includes: reading, by the computing unit, the key-value matrix in the second format from the key-value cache; and further including: using the calculation unit to inverse quantize the key value matrix in the second format based on an inverse quantization function constructed by the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in the first format, and input the key value matrix to an attention layer of the large-scale model. The large scale model inference method of claim 9.

11. the quantization function is used to perform a rounding operation on a key value matrix in a first format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in a second format after quantization; the inverse quantization function is used to perform floating-point operations on the key value matrix in the second format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in the first format after inverse quantization; The large scale model inference method of claim 10.

12. A storage device for quantization parameters, comprising: a statistical module for statistically obtaining, by a calculation unit of the processor, statistical values ​​of the first quantization parameters of the large-scale model based on the reference data; a search module for using the calculation unit to search for search parameters that minimize a loss function in a search space based on the statistics of the first quantization parameter, as target search parameters, and calculating and obtaining target values ​​of the first quantization parameter and the second quantization parameter of the large-scale model corresponding to the target search parameters based on the target search parameters; a calculation module for storing the target value of the first quantization parameter and the target value of the second quantization parameter in a memory by the calculation unit; Storage of quantization parameters.

13. The target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a large-scale model inference process, and are used to quantize a first-format key-value matrix required for an attention layer of the large-scale model into a second-format key-value matrix; wherein the key-value matrix in the second format is stored in a key-value cache of the processor. The quantization parameter storage device according to claim 12 .

14. the target value of the first quantization parameter and the target value of the second quantization parameter can be read from the memory to the processor in a large scale model inference process, and used to dequantize the second format key value matrix read from the key value cache into a first format key value matrix; Here, the dequantized key value matrix is ​​used as the input feature of the attention layer. The quantization parameter storage device according to claim 13 .

15. The statistics module: Statistically obtaining an average minimum value and an average maximum value of the first quantization parameter by averaging the reference data; and statistically obtaining a minimum value of the absolute maximum values ​​and a maximum value of the absolute maximum values ​​of the first quantization parameter by using an absolute maximum value for the reference data. The quantization parameter storage device according to claim 12 .

16. The search module: a first calculation submodule for calculating candidate values ​​of the first quantization parameter and candidate values ​​of the second quantization parameter based on an average minimum value, a minimum absolute maximum value, an average maximum value, a maximum absolute maximum value, and search parameters in the search space; a search submodule for calculating the value of the loss function based on the candidate values ​​of the first quantization parameter and the candidate values ​​of the second quantization parameter corresponding to all search parameters in a search space and the key value matrix of the reference data, and searching for and obtaining search parameters that minimize the value of the loss function as target search parameters; a second calculation sub-module for calculating, based on the target search parameter, a target value of the first quantization parameter and a target value of the second quantization parameter corresponding to the target search parameter; The quantization parameter storage device according to claim 15 .

17. The first calculation sub-module: calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a search parameter in the search space; calculating a maximum value of the first quantization parameter based on the average maximum, the maximum value of the absolute maximum, and a search parameter in the search space; calculating candidate values ​​of the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; and calculating candidate values ​​of the second quantization parameter based on the minimum value of the first quantization parameter and the candidate values ​​of the first quantization parameter. The quantization parameter storage device according to claim 16 .

18. The first calculation sub-module: calculating a minimum value of the first quantization parameter based on the average minimum value, the minimum of the absolute maximum values, and a first search parameter in a first search space; calculating a maximum value of the first quantization parameter based on the average maximum, the maximum of the absolute maximum values, and a second search parameter in a second search space; calculating candidate values ​​of the first quantization parameter based on a maximum value of the first quantization parameter and a minimum value of the first quantization parameter; and calculating candidate values ​​of the second quantization parameter based on the minimum value of the first quantization parameter and the candidate values ​​of the first quantization parameter. The quantization parameter storage device according to claim 16 .

19. the loss function is determined based on a quantization function and an inverse quantization function; the quantization function is used to perform a rounding operation on a key value matrix of a first format of the reference data based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix of a second format after quantization; the inverse quantization function is used to perform floating-point arithmetic on the quantized key value matrix in the second format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain the inverse quantized key value matrix in the first format. The quantization parameter storage device according to any one of claims 16 to 18.

20. A large-scale model inference device, comprising: a processing module for processing the input data of the large scale model with a computational unit of the processor to obtain a key-value matrix in a first format; a first reading module for reading, by the computing unit, a target value of a first quantization parameter and a target value of a second quantization parameter from a memory, the target value of the first quantization parameter and the target value of the second quantization parameter being stored in the memory before a large scale model inference process using the method of claim 1; a quantization module for quantizing the first-format key value matrix according to a quantization function established by the calculation unit using the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a second-format key value matrix; a storage module for storing, by the computation unit, the key-value matrix in the second format in a key-value cache of the processor; Large-scale model inference device.

21. The large-scale model inference device a second reading module for reading, by the computing unit, the key-value matrix in the second format from the key-value cache; and an inverse quantization module for inverse quantizing the second-format key value matrix based on an inverse quantization function constructed by the calculation unit using the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a first-format key value matrix and input the first-format key value matrix to an attention layer of the large-scale model.

21. The large scale model inference apparatus according to claim 20.

22. the quantization function is used to perform a rounding operation on a key value matrix in a first format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in a second format after quantization; the inverse quantization function is used to perform floating-point operations on the key value matrix in the second format based on the target value of the first quantization parameter and the target value of the second quantization parameter to obtain a key value matrix in the first format after inverse quantization; 22. The large scale model inference apparatus according to claim 21.

23. at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the method of any one of claims 1 to 11. Electronic devices.

24. A non-transitory computer readable storage medium having stored thereon computer instructions that cause a computer to perform the method of any one of claims 1 to 11.

25. A program for implementing the method of any one of claims 1 to 11 when executed by a processor in a computer.

Images