KV cache data quantification device and method

Abstract

The invention provides KV cache data quantization equipment, which comprises a precision decision interface, a compressor and a storage interface, and is characterized in that the precision decision interface is used for determining a quantization precision mark of each token in combination with token quantization difficulty, and generating a mixed quantization precision instruction according to the quantization precision marks corresponding to all the tokens; the compressor is used for acquiring a floating point Key value and / or a floating point Value value corresponding to each token, quantizing the floating point Key value and / or the floating point Value value based on a mixed quantization precision instruction issued by the precision decision interface, and generating metadata with a scaling factor and a zero point of each token and corresponding quantization data; and the storage interface is used for combining the quantized data corresponding to the quantized precision marks with the low-precision bit width in pairs, associating the quantized data with the metadata through an address mapping table, and separately storing the quantized data and the metadata. According to the invention, the storage space occupied by the KV cache can be reduced, and the reasoning speed of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, in particular to a KV cache data quantization device and method. BACKGROUND

[0002] In the inference process of a large language model (LLM), a KV cache (Key-Value Cache) mechanism is used to cache the keys and values calculated by the intermediate layer of the attention mechanism, so that the model can directly access these cached results when generating subsequent outputs, without the need to recalculate, thereby saving a large amount of computing resources and speeding up the model inference. However, as the length of the input text increases, the storage space required for the cached keys and values increases by a large proportion, thereby slowing down the model inference. SUMMARY

[0003] The embodiments of the present application provide a KV cache data quantization device, which can solve the technical problem of a large proportion of increase in the storage space required for the KV cache as the length of the input text increases, and improve the inference speed of the model.

[0004] In a first aspect, the embodiments of the present application provide a KV cache data quantization device, comprising a precision decision interface, a compressor and a storage interface,

[0005] The precision decision interface is configured to determine the quantization precision mark of each token in combination with the token quantization difficulty, and generate a hybrid quantization precision instruction according to the quantization precision marks corresponding to all tokens.

[0006] The compressor is configured to obtain the floating-point Key value and / or floating-point Value value corresponding to each token, quantize the floating-point Key value and / or floating-point Value value based on the hybrid quantization precision instruction issued by the precision decision interface, generate metadata with the scaling factor and zero point of each token, and corresponding quantization data.

[0007] The storage interface is configured to merge the quantization data corresponding to the quantization precision marks of low precision bit width two by two, associate the quantization data with the metadata through an address mapping table, and store the quantization data and the metadata separately.

[0008] Optionally, the precision decision interface is further configured to:

[0009] For each sequence to be quantized in each attention head:

[0010] Based on the outlier distribution of floating-point key and value values ​​in the sequence to be quantized, and / or, quantize the impact of the sequence to be quantized on the attention calculation results, determine the quantization difficulty corresponding to each token in the sequence to be quantized, and generate the corresponding quantization precision marker based on the corresponding quantization difficulty. The number of tokens in the sequence to be quantized is determined according to the seq_len value, and the dimension of each token is determined according to the head_dim value.

[0011] Optionally, the precision decision interface is also used for,

[0012] The upper and lower bounds of the floating-point key and floating-point value are calculated using the quartile method. Based on the upper and lower bounds, the deviation distances of outliers in the floating-point key and floating-point value are determined. The sum of the two deviation distances is taken as the quantization difficulty corresponding to the current token.

[0013] Optionally, the precision decision interface is also used for,

[0014] For each token, sort all floating-point key values ​​in the token in ascending order, and obtain the first difference between the upper quartile and the lower quartile of all floating-point key values.

[0015] The difference between the following quartiles and 1.5 times the first difference is used as the lower limit of the floating-point Key value.

[0016] The sum of the above quartiles and 1.5 times the first difference is used as the upper limit of the floating-point Key value;

[0017] Floating-point key values ​​outside the upper and lower limits of floating-point key values ​​are considered outliers. The deviation distance between the first floating-point key value above the upper limit and the upper limit is calculated, and the deviation distance between the second floating-point key value below the lower limit and the lower limit is calculated.

[0018] The maximum value among all deviation distances is taken as the deviation distance of the outlier value of the floating-point Key value;

[0019] For each token, sort all floating-point values ​​in the token in ascending order, and obtain the first difference between the upper quartile and the lower quartile of all floating-point values.

[0020] The difference between the following quartiles and 1.5 times the first difference is used as the lower limit of the floating-point value.

[0021] The sum of the above quartiles and 1.5 times the first difference is used as the upper limit of the floating-point value;

[0022] Floating-point values ​​outside the upper and lower limits of floating-point values ​​are considered outliers. The deviation distance between the first floating-point value above the upper limit and the upper limit is calculated, and the deviation distance between the second floating-point value below the lower limit and the lower limit is calculated.

[0023] The maximum value among all deviations is taken as the deviation distance of the outlier floating-point Value.

[0024] Optionally, the precision decision interface is also used for,

[0025] Using a low-precision bit width, the floating-point Key value and the floating-point Value value are quantized to obtain the first low-precision quantization result;

[0026] Calculate the mean square error between the floating-point key value and the first quantized key value in the first low-precision quantization result, and the mean square error between the floating-point value and the first quantized value in the first low-precision quantization result;

[0027] The sum of the two mean squared errors is used as the quantization difficulty corresponding to the current token.

[0028] Optionally, the precision decision interface is also used for,

[0029] Obtain the floating-point attention score matrix A corresponding to the floating-point key and floating-point value. float ;

[0030] By using a low-precision bit width, the floating-point Key value is quantized to obtain a second low-precision quantization result;

[0031] Based on the floating-point query value and the second quantization key value in the second low-precision quantization result, the quantization attention score matrix A is calculated. quant ;

[0032] Calculate the floating-point attention score matrix A column by column. float With the quantized attention score matrix A quant The difference is used to calculate the quantization difficulty of the corresponding token, and the sum of the differences of each element in the column is used as the quantization difficulty.

[0033] Optionally, the precision decision interface is also used for,

[0034] Retrieves the floating-point key, floating-point value, and floating-point query value, and the corresponding floating-point precision attention output matrix Z. float ;

[0035] Using a low-precision bit width, the floating-point Key value and the floating-point Value value are quantized to obtain a third low-precision quantization result;

[0036] Based on the floating-point query value and the quantization key and value values ​​from the third low-precision quantization result, the quantization attention output matrix Z is calculated. quant ;

[0037] The attention output matrix Z is calculated line by line for floating-point precision. float With the quantized attention output matrix Z quant The difference is used to calculate the quantification difficulty of the corresponding token, and the sum of the differences of each row element is used as the quantification difficulty.

[0038] Optionally, the precision decision interface is also used for,

[0039] Sort all tokens according to quantization difficulty, set the quantization precision flag of the top K tokens to high precision bit width, set the quantization precision flag of the remaining tokens to low precision bit width, and generate mixed quantization precision instructions for all tokens.

[0040] Optionally, the precision decision interface is also used for,

[0041] For each attention head, determine the N quantization difficulties of the N tokens in that attention head, and set an initial ratio threshold;

[0042] Calculate the sum of the quantization difficulties of N quantization methods.

[0043] Calculate the standard deviation of N quantization difficulties.

[0044] Based on the normalized sum, the normalized standard deviation, and the initial ratio threshold, the high-precision bit width ratio of the attention head is determined.

[0045] The K value is determined based on the N value and the high-precision bit width ratio.

[0046] Optionally, the compressor is also used for,

[0047] For each token, the scaling factor and zero point of the Key vector are determined based on the difference between the maximum and minimum values ​​among the head_dim floating-point Key values ​​in the Key vector. Similarly, the scaling factor and zero point of the Value vector are determined based on the difference between the maximum and minimum values ​​among the head_dim floating-point Value values ​​in the Value vector.

[0048] scale = (max - min) / (2) n -1);

[0049] zero_point=round(min / scale);

[0050] Where n represents high-precision bit width or low-precision bit width, scale represents scaling factor, zero_point represents zero point, max represents the maximum value of the key value or the maximum value of the value, and min represents the minimum value of the key value or the minimum value of the value.

[0051] Optionally, the quantization equipment may also include a decompressor.

[0052] The decompressor, in each decompression step, retrieves the address mapping table and the quantization precision flags corresponding to each token from the previous decompression step. For each token,

[0053] The compressed data packet containing the metadata of the token and the corresponding compressed data packet containing the quantized data are read from the address mapping table.

[0054] If the quantization precision marker corresponding to the token is low precision bit width, then each byte of the compressed data packet in the quantization data is split into two 4-bit quantization data; if the quantization precision marker corresponding to the token is high precision bit width, then the compressed data packet of the quantization data is not split, and the quantization data is obtained directly.

[0055] The metadata is read from the compressed metadata package. Based on the scaling factor and zero point of the Key and Value values ​​in the metadata, and combined with the corresponding quantization precision marker, the quantized data is dequantized to obtain floating-point Key values ​​and / or floating-point Value values ​​of type FP16, and the floating-point Key values ​​and / or floating-point Value values ​​are output.

[0056] Secondly, embodiments of this application provide a method for quantizing KV cached data, including:

[0057] The precision decision interface is called to determine the quantization precision tag for each token based on the token quantization difficulty, and a mixed quantization precision instruction is generated based on the quantization precision tags corresponding to all tokens.

[0058] Call the compressor to obtain the floating-point key value and / or floating-point value in each token, quantize the floating-point key value and / or floating-point value based on the mixed quantization precision instruction issued by the precision decision interface, and generate metadata with the scaling factor and zero point of each token, as well as the corresponding quantization data;

[0059] The storage interface is called to merge the quantization data corresponding to the low-precision quantization precision markers in pairs, associate the quantization data with the metadata through an address mapping table, and store the quantization data and metadata separately.

[0060] This application embodiment determines the corresponding quantization precision flag based on the quantization difficulty of each token through a precision decision interface, generates a hybrid quantization precision instruction carrying the quantization precision flag, and the compressor quantizes the floating-point key value and / or floating-point value according to the hybrid quantization precision instruction to obtain quantized data, as well as generate metadata with the scaling factor and zero point of each token. The storage interface stores the acquired quantized data and metadata separately, using the token's quantization difficulty as the basis for quantization allocation of the floating-point key value and / or floating-point value, thereby improving the performance of model quantization, greatly reducing the memory space occupied by the key-value cache, improving the read and write efficiency of the key-value cache, and helping to improve the model inference speed. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of the first system of the KV cache data quantization device provided in the embodiments of this application;

[0063] Figure 2 This is a schematic diagram of the floating-point attention score matrix provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the quantized attention score matrix provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the difference matrix provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the second system of the KV cache data quantization device provided in the embodiments of this application;

[0067] Figure 6 This is a flowchart illustrating the KV cache data quantization method provided in the embodiments of this application. Detailed Implementation

[0068] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0069] Please refer to Figure 1This application provides a KV cache data quantization device, which includes a precision decision interface 11, a compressor 12, and a storage interface 13. The precision decision interface 11 communicates with the compressor 12, and the storage interface 13 communicates with the compressor 12.

[0070] Large-scale language models typically employ a Transformer architecture, which consists of multiple layers of Transformer networks. The input text data is tokenized, yielding individual tokens. Within each Transformer layer, an attention mechanism performs a linear transformation on each token, generating corresponding Key, Value, and Query values. If the Transformer layer includes multiple attention heads, each attention head generates the corresponding Key, Value, and Query values ​​for each token. The Key and Value values ​​are cached during inference for use in subsequent steps.

[0071] The precision decision interface 11 determines the quantization precision flag for each token based on the token quantization difficulty, and generates a mixed quantization precision instruction based on the quantization precision flags corresponding to all tokens. For example, the token quantization difficulty is used to determine the quantization precision of the key and / or value values ​​corresponding to the token. This includes determining which tokens use high-precision bit width (e.g., 8-bit quantization) for their key and / or value values, and which use low-precision bit width (e.g., 4-bit quantization) for their key and / or value values. The mixed quantization precision instruction carries the token ID and the corresponding quantization precision flag. For example, the quantization precision flag is 0 or 1, where 0 represents low-precision bit width (e.g., 4-bit quantization) and 1 represents high-precision bit width (e.g., 8-bit quantization). The token ID is used as the unique identifier of the token to determine the quantization precision flag corresponding to each token. For example, if the token ID is 1, and the corresponding quantization difficulty determines the quantization precision to be 8 bits, then the quantization precision flag for token ID 1 is set to 1.

[0072] Compressor 12 acquires the floating-point Key and / or floating-point Value corresponding to each token, and quantizes the floating-point Key and / or floating-point Value based on the mixed quantization precision instruction issued by precision decision interface 11, generating metadata with scaling factor and zero point for each token, as well as the corresponding quantized data. For example, the floating-point Key and / or floating-point Value are FP16 floating-point type data. Compressor 12 parses the mixed quantization precision instruction output by precision decision interface 11, acquires the token id and corresponding quantization precision flag in the mixed quantization precision instruction, and performs quantization operation on the floating-point Key and / or floating-point Value corresponding to the token id according to the quantization precision flag, obtaining the corresponding quantized data, i.e., the quantized Key and / or Value. Compressor 12 generates metadata based on the token id, quantization precision flag, scaling factor, and zero point of each token, and outputs the metadata and quantized data to storage interface 13.

[0073] Storage interface 13 merges the quantization data corresponding to the low-precision bit-width quantization precision markers in pairs, associates the quantization data with metadata through an address mapping table, and stores the quantization data and metadata separately. For quantization data represented by quantization precision markers as low-precision bit-width quantization data, it caches the data based on the pairwise merging method, stores the quantization data and metadata separately, and associates the quantization data and metadata through an address mapping table.

[0074] For example, for quantized data corresponding to low-precision quantization precision markers, the quantized data is packaged and stored in the cache by storing two 4-bit data points in one byte. For quantized data corresponding to high-precision quantization precision markers, the data is stored directly in the cache without merging. For instance, data quantized with 4 bits (low precision) occupies 4 bits of storage space per data point. Since the minimum storage space for each memory address is one byte, the quantized data is merged into one byte for storage to avoid wasting storage space. Data quantized with 8 bits (high precision) occupies 8 bits (one byte) of storage space per data point, so it can be stored directly without merging. When storing the metadata for each token, the token ID, quantization precision marker, scaling factor, and zero point are stored in the corresponding storage address.

[0075] For example, an address mapping table is constructed, where each row represents the storage address of the metadata and the storage address of the quantized data corresponding to a token ID.

[0076] In this embodiment, the precision decision interface 11 determines the corresponding quantization precision flag based on the quantization difficulty of each token, and generates a hybrid quantization precision instruction carrying the quantization precision flag. The compressor 12 quantizes the floating-point key value and / or floating-point value according to the hybrid quantization precision instruction to obtain quantized data, and generates metadata with the scaling factor and zero point of each token. The storage interface 13 stores the acquired quantized data and metadata separately, using the quantization difficulty of the token as the basis for quantization allocation of the floating-point key value and / or floating-point value. This greatly reduces the memory space occupied by the key-value cache, improves the read and write efficiency of the key-value cache, improves the model inference speed, and reduces the model deployment cost. By storing the metadata and quantized data separately, the metadata is small in size and suitable for storage in high-speed structured memory for fast access and operation, while the quantized data is large in size and suitable for separate storage in object storage, distributed storage systems, etc., further saving memory.

[0077] In one embodiment of this application, the precision decision interface 11 is specifically used for: for the sequence to be quantized in each attention head; based on the outlier distribution of floating-point key values ​​and floating-point value values ​​in the sequence to be quantized, and / or, quantizing the impact of the sequence to be quantized on the attention calculation result, determining the quantization difficulty corresponding to each token in the sequence to be quantized, and generating a corresponding quantization precision label based on the corresponding quantization difficulty, wherein the number of tokens in the sequence to be quantized is determined according to the seq_len value, and the dimension of each token is determined according to the head_dim value.

[0078] Each Transformer network layer includes at least one attention head. Within each attention head, a linear transformation is performed on each input token to obtain the corresponding floating-point Key and Value values. The tensor shapes of both the Key and Value vectors are represented as [batch_size, num_heads, seq_len, head_dim], where batch_size represents the number of sequences processed simultaneously by the large language model, num_heads represents the number of attention heads in the multi-head attention model, seq_len represents the length of each sequence (i.e., the number of tokens in each sequence), and head_dim represents the vector dimension mapped to each token in each attention head.

[0079] In this embodiment, the quantization difficulty for each token is determined based on the quantization difficulty of the Key and Value values ​​corresponding to each token. The determination of the quantization difficulty of the Key and Value values ​​is based on the Key and Value values ​​calculated in each attention head of each sequence. According to the tensor shape of the Key and Value values, the dimensions of batch_size and num_heads in the Key and Value values ​​can be disregarded; only the vectors of seq_len and head_dim need to be considered. Therefore, the sequence to be quantized in each attention head consists of data in two dimensions: seq_len and head_dim. This includes floating-point Key vectors of seq_len and head_dim, and floating-point Value vectors of seq_len and head_dim. For example, seq_len = 100 and head_dim = 128 indicates that there are 100 tokens in the sequence to be quantized, and each token corresponds to 128 floating-point Key values ​​and 128 floating-point Value values.

[0080] In this embodiment, based on the outlier distribution of floating-point Key and Value values ​​in the sequence to be quantized, and / or the impact of the sequence to be quantized on the attention calculation result, the quantization difficulty corresponding to each token in the sequence to be quantized is determined, and a corresponding quantization precision mark is generated based on the corresponding quantization difficulty. The quantization difficulty of Key and Value values ​​is used as the basis for quantization precision allocation. Considering the error brought by quantization to the KV cache itself, its quantization effect is better.

[0081] In one embodiment of this application, the precision decision interface 11 is specifically used to: calculate the upper and lower limits of the floating-point key value and the floating-point value respectively using the quartile method; based on the upper and lower limits, determine the deviation distance of the outlier value of the floating-point key value and the deviation distance of the outlier value of the floating-point value; and use the sum of the two deviation distances as the quantization difficulty corresponding to the current token.

[0082] In this embodiment, the quartile method is used to determine the data distribution deviation of floating-point key values ​​and floating-point value values ​​through the statistical distribution of data, thereby determining the token quantization difficulty. The quartile method is more robust to outliers and is not easily affected by outliers, resulting in better quantization performance. It can also effectively compress the KV cache size and reduce memory usage.

[0083] Interquartiles, also known as quartiles, are statistically significant values ​​that fall at one of the three dividing points after all values ​​have been arranged in ascending order into four equal parts. Each quartile represents 25% of the total data. The lower quartile (Q1) is the 25th percentile of all values ​​arranged in ascending order, and the upper quartile (Q3) is the 75th percentile.

[0084] Specifically, for each token, all floating-point key values ​​in that token are arranged in ascending order. The first difference (IQR) between the upper quartile (Q3) and the lower quartile (Q1) of all floating-point key values ​​is obtained, denoted as IQR = Q3 - Q1. The difference between the lower quartile (Q1) and 1.5 times the first difference (IQR) is used as the lower limit of the floating-point key value, denoted as Q1 - 1.5 × IQR. The sum of the upper quartile (Q3) and 1.5 times the first difference (IQR) is used as the upper limit of the floating-point key value, denoted as Q3 + 1.5 × IQR. Floating-point key values ​​outside the upper and lower limits are considered outliers. The deviation distance between the first floating-point key value above the upper limit and the upper limit, and the deviation distance between the second floating-point key value below the lower limit and the lower limit are calculated. The maximum value among all deviation distances is used as the deviation distance of the outlier. For floating-point key values ​​between the upper and lower limits, the corresponding deviation distance is set to 0.

[0085] Specifically, for each token, all floating-point values ​​in the token are arranged in ascending order. The first difference between the upper and lower quartiles of all floating-point values ​​is obtained. The difference between the lower quartile and 1.5 times the first difference is used as the lower limit of the floating-point value. The sum of the upper quartile and 1.5 times the first difference is used as the upper limit of the floating-point value. Floating-point values ​​outside the upper and lower limits are considered outliers. The deviation distance between the first floating-point value above the upper limit and the upper limit is calculated, and the deviation distance between the second floating-point value below the lower limit and the lower limit is calculated. The maximum value among all deviation distances is used as the deviation distance of the outlier. For floating-point values ​​between the upper and lower limits, the corresponding deviation distance is set to 0.

[0086] In one embodiment of this application, the precision decision interface 11 is further configured to: quantize a floating-point Key value and a floating-point Value value using a low-precision bit width to obtain a first low-precision quantization result, the first low-precision quantization result including a first quantized Key value and a first quantized Value value; calculate the mean square error between the floating-point Key value and the first quantized Key value in the first low-precision quantization result, and the mean square error between the floating-point Value value and the first quantized Value value in the first low-precision quantization result; and use the sum of the two mean square errors as the quantization difficulty corresponding to the current token. For example, the low-precision bit width is 4 bits.

[0087] In one embodiment of this application, the precision decision interface 11 is further used to: obtain the floating-point attention score matrix A corresponding to the floating-point Key value and the floating-point Query value (query value). float Using a low-precision bit width, the floating-point key value is quantized to obtain a second low-precision quantization result; based on the floating-point query value and the second quantization key value in the second low-precision quantization result, the quantization attention score matrix A is calculated. quant Calculate the floating-point attention score matrix A column by column. float With the quantized attention score matrix A quant The difference is used to calculate the quantization difficulty of the corresponding token, and the sum of the differences of each element in the column is used as the quantization difficulty.

[0088] In this embodiment, the distribution offset caused by key / value quantization to the attention score is determined by the difference between the floating-point attention score matrix and the quantized attention score matrix. The quantization difficulty of the token is determined based on the distribution offset, and the quantization operation of floating-point key value / floating-point value is realized. This not only considers the error brought by quantization to the KV cache itself, but also the error brought to the subsequent attention score. Its quantization performance is better and helps to reduce the overall quantization loss of the model.

[0089] For example, the corresponding floating-point attention score matrix A is calculated based on the floating-point key value and the floating-point query value. float Floating-point attention score matrix A float Represented as:

[0090]

[0091] Where Q represents the floating-point Query value, and K1 represents the floating-point Key value. The dimension representing the key value. This represents the scaling factor.

[0092] For example, the low-precision bit width is 4 bits.

[0093] For example, the quantization attention score matrix A is calculated based on the floating-point query value and the second quantization key value in the second low-precision quantization result. quant Quantification of attention score matrix A quant Represented as:

[0094]

[0095] Where Q represents the floating-point query value, and K2 represents the second quantized key value. The dimension representing the key value. This represents the scaling factor.

[0096] The floating-point key vector, floating-point value vector, and floating-point query vector have the shape [seq_len, head_dim]. The floating-point attention score matrix A is calculated according to the above formula. float Quantized attention score matrix A quant The dimension is [seq_len, seq_len], which is the floating-point attention score matrix A. float The element in the i-th column represents the floating-point attention score corresponding to the i-th token, and the quantized attention score matrix A quant The elements in the i-th column represent the quantized attention score corresponding to the i-th token. Calculate the floating-point attention score matrix A. float The elements of the i-th column and the quantized attention score matrix A quant The difference between each element in the i-th column is used as the sum of the differences in the i-th column to obtain the quantization difficulty of the corresponding i-th token, thus obtaining the quantization difficulty of each token.

[0097] For example, seq_len is assumed to be 5, indicating that there are 5 tokens. Figure 2 As shown, the floating-point attention score matrix A calculated based on the above formula is... float This is a 5x5 matrix, where each column represents the floating-point attention score of the corresponding token. For example... Figure 3 The quantized attention score matrix A shown is calculated based on the above formula. quant This is a 5x5 matrix, where each column represents the quantized attention score of the corresponding token. The calculated floating-point attention score matrix A... float With the quantized attention score matrix A quant The absolute value of the difference between each corresponding element in the matrix is ​​used to obtain the following: Figure 4 The difference matrix shown is obtained by adding the elements of each column to get a one-dimensional vector: {0.18, 0.16, 0.33, 0.24, 0.19}, which corresponds to the quantization difficulty of each token.

[0098] In one embodiment of this application, the precision decision interface 11 is specifically used to: obtain the floating-point Key value, floating-point Value value, and floating-point Query value, and the corresponding floating-point precision attention output matrix Z. float Using a low-precision bit width, the floating-point key and value values ​​are quantized to obtain a third low-precision quantization result. Based on the floating-point query value and the quantized key and value values ​​in the third low-precision quantization result, the quantization attention output matrix Z is calculated. quant The attention output matrix Z is calculated line by line for floating-point precision. float With the quantized attention output matrix Z quant The difference is calculated, and the sum of the differences of each element in each row is used as the quantization difficulty of the corresponding token, thus obtaining the quantization difficulty of each token.

[0099] In this embodiment, the distribution offset caused by Key / Value quantization to the attention output is determined by the difference between the floating-point precision attention output matrix and the quantized attention output matrix. The quantization difficulty of the token is determined based on the distribution offset, and the quantization operation of floating-point Key / Value values ​​is realized. This not only considers the error brought by quantization to the KV cache itself, but also the error brought to subsequent attention output. Its quantization performance is better and helps to reduce the overall quantization loss of the model.

[0100] For example, the attention output matrix Z with floating-point precision is calculated based on the floating-point key value, floating-point value, and floating-point query value. float The attention output matrix Z with floating-point precision float Represented as:

[0101]

[0102] Where Q represents the floating-point Query value, K1 represents the floating-point Key value, and V1 represents the floating-point Value value. The dimension representing the key value. This represents the scaling factor.

[0103] For example, the low-precision bit width is 4 bits.

[0104] For example, the quantized attention output matrix Z is calculated based on the floating-point query value, the quantized key value, and the quantized value. quant Quantization attention output matrix Z quant Represented as:

[0105]

[0106] Where Q represents the floating-point query value, K2 represents the quantized key value, and V2 represents the quantized value. The dimension representing the key value. This represents the scaling factor.

[0107] The floating-point key vector, floating-point value vector, and floating-point query vector have shapes of [seq_len, head_dim]. The attention output matrix Z with floating-point precision is calculated according to the above formula. float And quantized attention output matrix Z quant The shape of the attention output matrix Z is [seq_len, head_dim], which is a floating-point precision matrix. float The elements in the i-th row represent the floating-point precision attention output value corresponding to the i-th token, and the quantization attention output matrix Z quant The elements in the i-th row represent the quantized attention output value corresponding to the i-th token. The floating-point precision attention output matrix Z is calculated. float The elements in the i-th row and the quantized attention output matrix Z quantt The difference between each element in the i-th row is used as the sum of the differences in the i-th row as the quantization difficulty of the corresponding i-th token, thus obtaining the quantization difficulty of each token.

[0108] In one embodiment of this application, the precision decision interface 11 is further configured to sort all tokens according to quantization difficulty, set the quantization precision flags of the top K sorted tokens to high precision bit width, set the quantization precision flags of the remaining tokens to low precision bit width, and generate mixed quantization precision instructions corresponding to all tokens.

[0109] For example, a fixed percentage threshold is set, and the value of K is determined based on this percentage threshold and the total number of tokens. For instance, if the percentage threshold is set to 40%, the total number of tokens is 100, and the value of K is 40, all tokens are sorted from largest to smallest according to quantization difficulty. Based on the sorted tokens, the quantization precision of the first 40 tokens is set to high precision bit width, and the corresponding quantization precision is marked as 1. The remaining 60 tokens are set to low precision bit width, and the corresponding quantization precision is marked as 0. A mixed quantization precision instruction is generated for all tokens.

[0110] For example, a difficulty threshold is set based on the quantization difficulty of each token. For instance, the difficulty threshold can be set to the mean or median of the quantization difficulty of each token. For each token, if the quantization difficulty of the token is greater than or equal to the difficulty threshold, the quantization precision flag of the token is set to high precision bit width; if the quantization difficulty of the token is less than the difficulty threshold, the quantization precision flag of the token is set to low precision bit width. The difficulty threshold is dynamically determined based on the quantization difficulty of each token.

[0111] For example, for each attention head, determine the N quantization difficulties of the N tokens in that attention head, and set an initial ratio threshold K1; calculate the sum S of the N quantization difficulties. sum1 If S sum1 The larger the value, the higher the overall quantization difficulty of the tokens in that attention head; calculate the standard deviation S of the N quantization difficulties. std1 If S std1 The larger the value, the more concentrated the overall quantization difficulty distribution of the tokens in that attention point, meaning that the quantization difficulty of a few tokens is higher than that of other tokens; summing S sum1 With standard deviation S std1 Normalization is performed to ensure that the data ranges of the two are consistent; based on the normalized sum S sum2 Normalized standard deviation S std2 And an initial ratio threshold K1 is used to determine the high-precision bit width ratio K2 of the attention head; the value of K is determined based on the N value and the high-precision bit width ratio K2. The formula for calculating the high-precision bit width ratio K2 is as follows:

[0112] K2=(S sum2 +S std2 )·K1;

[0113] In practical applications, any of the above-mentioned implementation methods for K values ​​can be selected as needed.

[0114] In one embodiment of this application, the compressor 12 is specifically used to: for each token, determine the scaling factor and zero point of the Key vector based on the difference between the maximum and minimum values ​​among the head_dim floating-point Key values ​​in the Key vector; and determine the scaling factor and zero point of the Value vector based on the difference between the maximum and minimum values ​​among the head_dim Value values ​​in the Value vector, wherein the calculation formulas for the scaling factor and the zero point are expressed as follows:

[0115] scale = (max - min) / (2) n -1);

[0116] zero_point=round(min / scale);

[0117] Where scale represents the scaling factor, zero_point represents the zero point, max represents the maximum value in the key vector or the maximum value in the value vector, min represents the minimum value in the key vector or the minimum value in the value vector, and n represents the high-precision bit width or the low-precision bit width. For example, n=4 represents the low-precision bit width, and n=8 represents the high-precision bit width.

[0118] In this embodiment, for each token's floating-point Key vector / floating-point Value vector to be quantized, the compressor 12 parses the token id and the corresponding quantization precision flag in the mixed quantization precision instruction. Based on the quantization precision flag corresponding to the token id, the quantization precision bit width corresponding to the token is determined. For example, a quantization precision flag of 0 indicates a low precision bit width (e.g., 4-bit quantization), and a quantization precision flag of 1 indicates a high precision bit width (e.g., 8-bit quantization). Based on the above scaling factor and zero-point calculation formula, the scaling factor and zero point of the Key vector and the scaling factor and zero point of the Value vector can be obtained. Quantization operations are then performed on the floating-point Key vector / floating-point Value vector to obtain the quantized Key value and / or Value value. The quantization operation is expressed as follows:

[0119] int_val=round((FP16_val-zero_point) / scale);

[0120] Here, FP16_val represents the floating-point key value / floating-point value.

[0121] One embodiment of this application, such as Figure 5 As shown, the quantization device also includes a decompressor 14. In each decompression step, the decompressor 14 obtains the address mapping table and the quantization precision flag corresponding to each token in the previous decompression step. For each token,

[0122] The compressed data packet containing the metadata of the token and the corresponding compressed data packet containing the quantized data are read from the address mapping table.

[0123] If the quantization precision flag corresponding to the token is low precision bit width, then each byte of the compressed data packet in the quantization data is split into two 4-bit quantization data; if the quantization precision flag corresponding to the token is high precision bit width, then the compressed data packet of the quantization data is not split, and the quantization data is obtained directly.

[0124] Read the metadata from the compressed metadata package. Based on the scaling factor and zero point of the Key and Value values ​​in the metadata, and combined with the corresponding quantization precision marker, dequantize the quantized data to obtain floating-point Key values ​​and / or floating-point Value values ​​of type FP16. Output the floating-point Key values ​​and / or floating-point Value values ​​for model calculation.

[0125] For example, the dequantization operation is represented as:

[0126] FP16_val=int_val*scale+zero_point,

[0127] Wherein, FP16_val represents the floating-point key value / floating-point value of type FP16, int_val represents the quantized key value / value, scale represents the scaling factor of the key vector / value vector, and zero_point represents the zero point of the key vector / value vector.

[0128] For example, the following example illustrates the workflow of the compressor and decompressor in detail. Assume the input to the large language model is: "What's the weather like today?" The model needs to output: "Sunny turning cloudy". Each Chinese character corresponds to one token. The large language model outputs the four tokens "Sunny turning cloudy", requiring a total of four generation steps, with one token output each time. The generation process is shown in the table below:

[0129] Generation step Input token Output token 1st generation What's the weather like today? Sunny 2nd generation What's the weather like today? Sunny Change 3rd generation What's the weather like today? Sunny change Much 4th generation What's the weather like today? Sunny change much Cloudy

[0130] In the first generation step, based on the precision decision interface, compressor, and storage interface in the embodiments of this application, the floating-point key / value values ​​corresponding to the 8 tokens "How's the weather today?" are quantized, and the 8 quantized key / value pairs are stored. In the second generation step, the decompressor is called to dequantize the 8 quantized key / value pairs obtained in the first step into corresponding FP16 type floating-point key / value values ​​for model calculation. Since the second generation includes an additional token "Sunny" compared to the first generation, the floating-point key / value value corresponding to this new token "Sunny" needs to be quantized, and the quantized key / value values ​​need to be stored. Therefore, 9 quantized key / value pairs corresponding to 9 tokens need to be stored. In the third and fourth generation steps, the compression and decompression of the KV cache are implemented in the same way as described above.

[0131] like Figure 6As shown, this application provides a method for quantizing KV cached data, the method comprising:

[0132] S601, call the precision decision interface to determine the quantization precision mark of each token in combination with the token quantization difficulty, and generate a mixed quantization precision instruction based on the quantization precision marks corresponding to all tokens;

[0133] S602, call the compressor to obtain the floating-point key value and / or floating-point value in each token, quantize the floating-point key value and / or floating-point value based on the mixed quantization precision instruction issued by the precision decision interface, and generate metadata with the scaling factor and zero point of each token, as well as the corresponding quantization data;

[0134] S603 calls the storage interface to merge the quantization data corresponding to the low-precision quantization precision markers in pairs, associates the quantization data with the metadata through the address mapping table, and stores the quantization data and metadata separately.

[0135] This application provides an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. In implementation, the processor, communication interface, and memory may also communicate with each other using other connection methods besides the communication bus.

[0136] The memory can be used to store computer programs, which may include instructions and data to implement the steps of any of the KV cache data quantization methods described above. In the embodiments of this application, the memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. The memory may include hard disks and / or RAM.

[0137] The processor can be a general-purpose processor, which can be a processor that performs specific steps and / or operations by reading and executing a computer program (e.g., a computer program) stored in memory (e.g., a memory module). The general-purpose processor may use data stored in memory (e.g., data in memory) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, the processor can also be a special-purpose processor, which can be a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, the processor can be a combination of multiple processors, such as a multi-core processor.

[0138] Communication interfaces can include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within a network device, as well as interfaces used for interconnecting network devices with other devices (e.g., network devices). Communication networks can be Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. Communication interfaces can be modules, circuits, transceivers, or any device capable of enabling communication.

[0139] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0140] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0141] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0142] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0143] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0144] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A KV cached data quantization device, characterized in that, This includes a precision decision interface, a compressor, and a storage interface. The precision decision interface is used to determine the quantization precision tag of each token by combining the token quantization difficulty, and to generate a mixed quantization precision instruction based on the quantization precision tags corresponding to all tokens. The compressor is used to obtain the floating-point key value and / or floating-point value corresponding to each token, quantize the floating-point key value and / or floating-point value based on the hybrid quantization precision instruction issued by the precision decision interface, and generate metadata with the scaling factor and zero point of each token, as well as the corresponding quantization data. The storage interface is used to merge the quantization data corresponding to the low-precision bit-width quantization precision markers in pairs, associate the quantization data with the metadata through an address mapping table, and store the quantization data and the metadata separately.

2. The quantization device as described in claim 1, characterized in that, The accuracy decision interface is also used for: For the sequence to be quantized in each attention head: Based on the outlier distribution of the floating-point Key and Value values ​​in the sequence to be quantized, and / or, quantize the impact of the sequence to be quantized on the attention calculation result, determine the quantization difficulty corresponding to each token in the sequence to be quantized, and generate a corresponding quantization precision marker based on the corresponding quantization difficulty, wherein the number of tokens in the sequence to be quantized is determined according to the seq_len value, and the dimension of each token is determined according to the head_dim value.

3. The quantization device as described in claim 1, characterized in that, The accuracy decision interface is also used for, The upper and lower limits of the floating-point key and floating-point value are calculated using the quartile method. Based on the upper and lower limits, the deviation distances of outliers of the floating-point key and floating-point value are determined. The sum of the two deviation distances is taken as the quantization difficulty corresponding to the current token.

4. The quantification device as described in claim 3, characterized in that, The accuracy decision interface is also used for, For each token, sort all floating-point key values ​​in the token in ascending order, and obtain the first difference between the upper quartile and the lower quartile of all floating-point key values. The difference between the lower quartile and 1.5 times the first difference is used as the lower limit of the floating-point Key value. The upper limit of the floating-point Key value is the sum of 1.5 times the difference between the upper quartile and the first value. Floating-point key values ​​outside the upper and lower limits of the floating-point key value are considered as outliers of the floating-point key value. The deviation distance between the first floating-point key value above the upper limit and the upper limit is calculated, and the deviation distance between the second floating-point key value below the lower limit and the lower limit is calculated. The maximum value among all deviation distances is taken as the deviation distance of the outlier value of the floating-point Key value; For each token, sort all floating-point values ​​in the token in ascending order, and obtain the first difference between the upper quartile and the lower quartile of all floating-point values. The difference between the lower quartile and 1.5 times the first difference is used as the lower limit of the floating-point Value. The upper limit of the floating-point Value is the sum of 1.5 times the difference between the upper quartile and the first value. Floating-point values ​​outside the upper and lower limits of the floating-point value are considered outliers of the floating-point value. The deviation distance between the first floating-point value above the upper limit and the upper limit is calculated, and the deviation distance between the second floating-point value below the lower limit and the lower limit is calculated. The maximum value among all deviations is taken as the deviation distance of the outlier floating-point Value.

5. The quantization device as described in claim 1, characterized in that, The accuracy decision interface is also used for, Using a low-precision bit width, the floating-point Key value and the floating-point Value value are quantized to obtain the first low-precision quantization result; Calculate the mean square error between the floating-point key value and the first quantization key value in the first low-precision quantization result, and the mean square error between the floating-point value and the first quantization value in the first low-precision quantization result; The sum of the two mean squared errors is used as the quantization difficulty corresponding to the current token.

6. The quantization device as described in claim 1, characterized in that, The accuracy decision interface is also used for: Obtain the floating-point attention score matrix A corresponding to the floating-point Key value and the floating-point Value value. float ; The floating-point Key value is quantized using a low-precision bit width to obtain a second low-precision quantization result; Based on the floating-point query value and the second quantization key value in the second low-precision quantization result, the quantization attention score matrix A is calculated. quant ; Calculate the floating-point attention score matrix A column by column. float With the quantized attention score matrix A quant The difference is used to calculate the quantization difficulty of the corresponding token, and the sum of the differences of each element in the column is used as the quantization difficulty.

7. The quantization device as described in claim 1, characterized in that, The accuracy decision interface is also used for: Obtain the floating-point Key value, the floating-point Value value, and the floating-point query value, and the corresponding floating-point precision attention output matrix Z. float ; Using a low-precision bit width, the floating-point Key value and the floating-point Value value are quantized to obtain a third low-precision quantization result; Based on the floating-point query value and the quantization key and value values ​​in the third low-precision quantization result, the quantization attention output matrix Z is calculated. quant ; The attention output matrix Z of the floating-point precision is calculated row by row. float With the quantized attention output matrix Z quant The difference is used to calculate the quantification difficulty of the corresponding token, and the sum of the differences of each row element is used as the quantification difficulty.

8. The quantization device as described in any one of claims 3-7, characterized in that, The accuracy decision interface is also used for, Sort all tokens according to quantization difficulty, set the quantization precision flag of the top K tokens to high precision bit width, set the quantization precision flag of the remaining tokens to low precision bit width, and generate mixed quantization precision instructions for all tokens.

9. The quantization device as described in claim 8, characterized in that, The accuracy decision interface is also used for, For each attention head, determine the N quantization difficulties of the N tokens in that attention head, and set an initial ratio threshold; Calculate the sum of the quantization difficulties of N quantization methods. Calculate the standard deviation of N quantization difficulties. Based on the normalized sum, the normalized standard deviation, and the initial ratio threshold, the high-precision bit width ratio of the attention head is determined. The K value is determined based on the N value and the high-precision bit width ratio.

10. The quantization device as described in claim 2, characterized in that, The compressor is also used for: For each token, the scaling factor and zero point of the key value are determined based on the difference between the maximum and minimum values ​​among the head_dim floating-point key values ​​in the key vector. Similarly, the scaling factor and zero point of the value value are determined based on the difference between the maximum and minimum values ​​among the head_dim floating-point value values ​​in the value vector. scale=(max-min) / (2 n -1); zero_point=round(min / scale); Where n represents high-precision bit width or low-precision bit width, scale represents scaling factor, zero_point represents zero point, max represents the maximum value of the key value or the maximum value of the value, and min represents the minimum value of the key value or the minimum value of the value.

11. The quantization device as described in claim 8, characterized in that, The quantization device further includes a decompressor, which is used to, in each decompression step, obtain an address mapping table and the quantization precision flag corresponding to each token in the previous decompression step, and for each token, read the compressed data packet of the token's metadata and the corresponding compressed data packet of quantization data according to the address mapping table. If the quantization precision marker corresponding to the token is low precision bit width, then each byte of the compressed data packet in the quantization data is split into two 4-bit quantization data; if the quantization precision marker corresponding to the token is high precision bit width, then the compressed data packet of the quantization data is not split, and the quantization data is obtained directly. The metadata is read from the compressed metadata package. Based on the scaling factor and zero point of the Key and Value values ​​in the metadata, and combined with the corresponding quantization precision marker, the quantized data is dequantized to obtain floating-point Key values ​​and / or floating-point Value values ​​of type FP16, and the floating-point Key values ​​and / or floating-point Value values ​​are output.

12. A method for quantizing KV cached data, characterized in that, include: The precision decision interface is called to determine the quantization precision tag for each token based on the token quantization difficulty, and a mixed quantization precision instruction is generated based on the quantization precision tags corresponding to all tokens. The compressor is invoked to obtain the floating-point key value and / or floating-point value in each token. Based on the hybrid quantization precision instruction issued by the precision decision interface, the floating-point key value and / or floating-point value are quantized to generate metadata with the scaling factor and zero point of each token, as well as the corresponding quantization data. The storage interface is called to merge the quantization data corresponding to the low-precision quantization precision markers in pairs, and the quantization data is associated with the metadata through an address mapping table. The quantization data and the metadata are stored separately.

Images