Key value processing method and device, text processing method and device, equipment and storage medium
By transforming and quantizing key values in a large language model, the accuracy problem caused by key-value caching quantization is solved, achieving higher model inference accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ALIBABA CLOUD COMPUTING CO LTD
- Filing Date
- 2024-11-25
- Publication Date
- 2026-05-26
AI Technical Summary
Key-value caching quantization leads to lower accuracy of inference results in large language models.
By converting multiple first key values into second key values with smaller data ranges and lower outlier degrees, and then numerically quantizing them, a third key value is obtained, which is then stored and dequantized to recover data close to the original.
It reduces the quantization error of key-value caching and improves the accuracy and efficiency of inference results in large language models.
Smart Images
Figure CN122087626A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and more specifically, to key-value processing methods, text processing methods, apparatus, devices, and storage media. Background Technology
[0002] Key-value caching (KV caching) is a common technique for optimizing the inference performance of large language models. In the self-attention mechanism of large language models, key-value caching is used to store already computed key-value pairs so that when reusing key-value pairs, the stored key-value pairs can be directly retrieved, avoiding repeated calculations and thus improving the inference speed of the model.
[0003] Key-value caching quantization is commonly used in related technologies to reduce the storage overhead of key-value caching. However, errors in key-value caching quantization can lead to lower accuracy in the inference results of large language models. Summary of the Invention
[0004] This application provides key-value processing methods, text processing methods, apparatus, devices, and storage media to address the problem that errors in key-value caching quantization can lead to low accuracy in reasoning results for large language models.
[0005] Firstly, this application proposes a key-value processing method, including:
[0006] In the key-value calculation process of the self-attention mechanism of the large language model, each of the multiple first key values is converted into a corresponding second key value. Each first key value includes a first key and a first value, and each second key value includes a second key and a second value. The data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. The outlier degree of the outlier values among the multiple second keys is smaller than the outlier degree of the corresponding outlier values among the multiple first keys, and the outlier degree of the outlier values among the multiple second values is smaller than the outlier degree of the corresponding outlier values among the multiple first values.
[0007] The second key value is numerically quantized to obtain the corresponding third key value, and the third key value is stored.
[0008] Secondly, this application proposes a text processing method, including:
[0009] Get the text to be processed;
[0010] The text to be processed is input into a pre-trained large language model to obtain the response output by the large language model. The large language model employs a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts the key-value processing method as described in any of the first aspects.
[0011] Thirdly, this application proposes a key-value processing apparatus, comprising:
[0012] The conversion module is used to convert each of a plurality of first key values into a corresponding second key value during the key value calculation process of the self-attention mechanism of the large language model. The first key value includes a first key and a first value, and the second key value includes a second key and a second value. The data range of the plurality of second keys is smaller than the data range of the plurality of first keys, and the data range of the plurality of second values is smaller than the data range of the plurality of first values. The outlier degree of the outlier values among the plurality of second keys is smaller than the outlier degree of the corresponding outlier values among the plurality of first keys, and the outlier degree of the outlier values among the plurality of second values is smaller than the outlier degree of the corresponding outlier values among the plurality of first values.
[0013] The quantization module is used to perform numerical quantization on the second key value to obtain the corresponding third key value, and to store the third key value.
[0014] Fourthly, this application proposes a text processing apparatus, comprising:
[0015] The acquisition module is used to acquire the text to be processed;
[0016] The processing module is used to input the text to be processed into a pre-trained large language model and obtain the response output by the large language model, wherein the large language model adopts a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts the key-value processing method as described in any of the first aspects.
[0017] Fifthly, this application provides an electronic device, including: a memory and a processor;
[0018] The memory is used to store computer programs;
[0019] The processor is configured to execute a computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method as described in either the first or second aspect.
[0020] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method as described in either the first or second aspect.
[0021] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.
[0022] The key-value processing method, text processing method, apparatus, device, and storage medium provided in this application convert each of a plurality of first key values into a corresponding second key value. Each first key value includes a first key and a first value, and each second key value includes a second key and a second value. The data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. The outlier degree of the outliers among the multiple second keys is smaller than the outlier degree of the corresponding outliers among the multiple first keys, and the outlier degree of the outliers among the multiple second values is smaller than the outlier degree of the corresponding outliers among the multiple first values. The second key values are then numerically quantized to obtain corresponding third key values, which are then stored. This method, by converting multiple first key values, narrows the data range of the key values and reduces the outlier degree of outliers. Since quantization is performed on relatively concentrated data, the quantization error is relatively small. Therefore, quantization based on the converted key values reduces the quantization error of the key-value cache and improves the accuracy of the inference results of large language models. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 1 ;
[0025] Figure 2 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 2 ;
[0026] Figure 3 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 3 ;
[0027] Figure 4 A flowchart illustrating a key-value processing method proposed in this application. Figure 4 ;
[0028] Figure 5 This is a schematic diagram of the structure of a key-value processing device provided in an embodiment of this application;
[0029] Figure 6 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;
[0030] Figure 7This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0032] To facilitate understanding of the embodiments of this application, the relevant terms involved in this application will be briefly explained below.
[0033] Key-value caching: A caching system used to store and quickly access key-value pairs of data, applicable to various data access scenarios that require high performance and low latency.
[0034] Numerical quantization is a method that saves space by transforming data. Quantization can significantly reduce the storage requirements and computational complexity of models, thereby improving inference speed and energy efficiency.
[0035] Outliers are values in a dataset that deviate significantly from other data points. Outliers are significantly different from other values in the dataset.
[0036] In large language models, self-attention mechanisms capture contextual information by calculating the similarity between queries, keys, and values. Key-value caching technology caches the calculated keys and values for reuse in subsequent inference processes, thereby avoiding redundant calculations and improving inference efficiency. Quantization of key-value caching is a method to save storage requirements for keys and values when caching them. For clarity and brevity, the key-value pairs described in the embodiments of this application are brief descriptions of keys and values.
[0037] The purpose of quantization is to reduce storage requirements and computational complexity, but the quantization process introduces certain errors. In related technologies, quantization of data with a relatively concentrated distribution results in relatively small quantization errors because the data is distributed relatively evenly across the quantization range, and the quantization range can effectively cover the data range. Conversely, if the data distribution is more dispersed, especially when outliers are present, the quantization error will increase. Therefore, existing key-value caching quantization methods have large quantization errors.
[0038] In view of this, this application proposes a key-value processing method, which converts each of a plurality of first key values into a corresponding second key value. Each first key value includes a first key and a first value, and each second key value includes a second key and a second value. The data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. The outlier degree of the outliers among the multiple second keys is smaller than the outlier degree of the corresponding outliers among the multiple first keys, and the outlier degree of the outliers among the multiple second values is smaller than the outlier degree of the corresponding outliers among the multiple first values. The second key values are then numerically quantized to obtain corresponding third key values, which are then stored. This method, by converting multiple first key values, narrows the data range of the key values and reduces the outlier degree of the outliers. Since quantization is performed on relatively concentrated data, the quantization error is relatively small. Therefore, quantization based on the converted key values reduces the quantization error of the key-value cache and improves the accuracy of the inference results of large language models.
[0039] It should be understood that the key-value processing method of this application embodiment can be used for key-value processing in any scenario where key-value caching can be implemented. The execution subject of this application embodiment can be any software and / or hardware that can implement the key-value processing method, such as any electronic device or apparatus that implements the key-value processing method of this application embodiment.
[0040] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.
[0041] Figure 1 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 1 In the key-value calculation process of the self-attention mechanism in a large language model, after obtaining the first key-value, as follows... Figure 1 As shown, the method may include, for example, the following steps:
[0042] S101. Convert each of the multiple first key values into its corresponding second key value.
[0043] The first key value includes a first key and a first value. Multiple first keys or multiple first values can be represented by any mathematical structure, such as a tensor, a matrix, a set, etc.
[0044] The second key value includes a second key and a second value. Multiple second keys, or multiple second values, can also be represented by any mathematical structure.
[0045] The data range of multiple second keys is smaller than the data range of multiple first keys, and the data range of multiple second values is smaller than the data range of multiple first values. Here, the data range refers to the range defined by the maximum and minimum values among the data; the data range of multiple second keys is the range defined by the largest and smallest second keys among the multiple second keys.
[0046] The outlier degree of outliers among multiple second-key outliers is less than that of their corresponding first-key outliers, and the outlier degree of outliers among multiple second-key outliers is less than that of their corresponding first-key outliers. The outlier degree refers to the degree of deviation of an outlier from other data points, and can be analyzed using outlier degree calculation methods such as the quartile range method and Mahalanobis distance. Taking any outlier among the first-key outliers as an example, the outlier degree of this outlier in the first-key set is calculated to obtain the first outlier degree. After transforming this outlier, the outlier degree of the transformed outlier in the second-key set is calculated to obtain the second outlier degree, which is less than the first outlier degree.
[0047] One possible implementation involves using exponential operations to convert each of the multiple first key values into a corresponding second key value. In this method, multiple first keys are exponentially operated on according to a pre-defined exponent to obtain multiple second keys. The data range of the resulting multiple second keys is smaller than the data range of the multiple first keys, and the outlier of a first key after exponential operation is less pronounced in the second keys than in the first keys. For example, if all first keys are values greater than 1, the exponent can be pre-defined as a number greater than 0 and less than 1, and the multiple first keys can be exponentially operated on to obtain multiple second keys. Optionally, in this method, if the multiple first keys include negative numbers, the sign of the multiple first keys can be determined, and the absolute values of the multiple first keys can be exponentially operated on to obtain multiple second keys.
[0048] Similarly, in this method, multiple first values can be exponentially calculated based on a pre-set exponent to obtain multiple second values. The data range of the resulting multiple second values is smaller than the data range of the multiple first values, and the outlier of the first value after exponential calculation is less pronounced in the second values than the outlier of the first value itself.
[0049] One feasible approach is to use logarithmic operations to convert each of the multiple first key values into a corresponding second key value. In this approach, the base of the logarithmic operation can be set based on the first key values, ensuring that after performing the logarithmic operation on the multiple first keys and values, the data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. Furthermore, the outlier degree of the outliers among the multiple second keys is less than the outlier degree of the corresponding outliers among the multiple first keys, and the outlier degree of the outliers among the multiple second values is less than the outlier degree of the corresponding outliers among the multiple first values.
[0050] It should be understood that the method for converting each of the multiple first key values into its corresponding second key value can be determined based on the data distribution of the first key values. This method is feasible as long as the data range of the converted second keys is smaller than the data range of the multiple first keys, the data range of the multiple second values is smaller than the data range of the multiple first values, the outlier degree of the outliers in the multiple second keys is smaller than the outlier degree of the corresponding outliers in the multiple first keys, and the outlier degree of the outliers in the multiple second values is smaller than the outlier degree of the corresponding outliers in the multiple first values. This application does not impose any restrictions on this. For example, if all the values in the first key values are greater than 1, the method of using reciprocals can also be used to convert each of the multiple first key values into its corresponding second key value.
[0051] S102. Quantize the second key value to obtain the corresponding third key value and store the third key value.
[0052] The third key value is obtained by quantizing the second key value. The third key value includes multiple third keys and third values. The third key value obtained after numerical quantization requires less storage space than the second key value.
[0053] Numerical quantization of the second key value can be performed using any numerical quantization method in related technologies. For example, it can be based on a fixed-point quantization method, converting a floating-point number into a fixed-width integer, typically 8 bits or less. Numerical quantization methods can also be symmetric or asymmetric. Symmetric quantization typically uses the minimum and maximum values of the quantization range to define the quantization range, which is symmetrical about zero. Asymmetric quantization can redefine a zero point, using the minimum, maximum, and zero points to define the mapping relationship from the original value to the quantized value. This application does not limit the numerical quantization method.
[0054] For example, when using a numerical quantization method that converts 16-bit floating-point quantization to 8-bit integers, the minimum boundary of the quantization range can be determined to be 0, and the maximum boundary to be 255. Based on the distribution of the quantization range and the original data, multiple second keys represented by 16-bit data can be mapped to multiple third keys represented by 8 bits, and multiple second values represented by 16-bit data can be mapped to multiple third values represented by 8 bits. Because the numerical quantization operation involves truncation or rounding of the key values to map them to a smaller quantization range, the corresponding third key value is obtained based on the numerical quantization and stored. Since the bit width of the third key value is smaller than that of the second key value, the space required to store the third key value is reduced compared to storing the second key value.
[0055] The key-value processing method of this application converts each of a plurality of first key values into a corresponding second key value. Each first key value includes a first key and a first value, and each second key value includes a second key and a second value. The data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. The outlier degree of the outliers among the multiple second keys is smaller than the outlier degree of the corresponding outliers among the multiple first keys, and the outlier degree of the outliers among the multiple second values is smaller than the outlier degree of the corresponding outliers among the multiple first values. The second key values are then numerically quantized to obtain corresponding third key values, which are then stored. This method, by converting multiple first key values, narrows the data range of the key values and reduces the outlier degree of outliers. Since numerical quantization is performed on relatively concentrated data, the quantization error is relatively small. Therefore, numerical quantization based on the converted key values reduces the quantization error of the key-value cache and improves the accuracy of the inference results of large language models.
[0056] After storing the third key value, in order to reuse the stored key value, the key value processing method provided in this application embodiment can also perform key value processing after obtaining the stored third key value to obtain a reusable key value. Figure 2 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 2 .like Figure 2 As shown, the specific steps can be performed as follows:
[0057] S201. Dequantize the third key value to obtain the dequantized third key value.
[0058] Dequantization is a method to restore numerically quantized data to a state close to the data before numerical quantization. The purpose of dequantization is to recover the data before numerical quantization, but because truncation or rounding operations occur during numerical quantization, there is a quantization error between the dequantized data and the data before numerical quantization.
[0059] Dequantization performs the opposite operation to the quantization method in step S102. The third key value includes a third key and a third value. Dequantization is performed on multiple third keys and multiple third values respectively to obtain the dequantized third key value.
[0060] S202. Convert the dequantized third key value into the corresponding fourth key value.
[0061] The operation of converting the dequantized third key value to the corresponding fourth key value is the inverse operation of converting each of the multiple first key values to the corresponding second key value; that is, it is the inverse operation of the conversion operation used in step S101. For example, if exponential operation is used to convert each of the multiple first key values to the corresponding second key value, then logarithmic operation is used to convert each of the multiple dequantized third key values to the corresponding fourth key value. Similarly, if logarithmic operation is used to convert each of the multiple first key values to the corresponding second key value, then exponential operation is used to convert each of the multiple dequantized third key values to the corresponding fourth key value.
[0062] The fourth key value includes a fourth key and a fourth value. The fourth key value is a key value that is recovered and close to the first key value. The obtained fourth key is a value that is recovered and close to the first key based on the third key, and the obtained fourth value is a value that is recovered and close to the first value based on the third value.
[0063] The key-value processing method in this application embodiment dequantizes the third key value to obtain the dequantized third key value, and converts the dequantized third key value into the corresponding fourth key value to obtain data close to the first key value. By reusing the obtained fourth key value, the key value is avoided from being calculated again, which can improve efficiency.
[0064] The following provides an example illustrating the specific steps of the key-value processing method proposed in this application. For instance, the method converts each of multiple first key values into its corresponding second key value, confirming the sign of the first key value and then converting it to the second key value based on its absolute value.
[0065] Figure 3 A flowchart illustrating a key-value processing method provided in this application embodiment. Figure 3 .like Figure 3 As shown, the key-value processing method specifically includes the following steps:
[0066] S301. Determine the sign of the elements in the first key value and determine the absolute value of the elements in the first key value.
[0067] The element is either the first key or the first value. Determine the sign of the element in the first key-value pair, that is, determine the sign of the first key and the sign of the first value.
[0068] Determine the sign of the elements in the first key value, i.e., extract the sign of the first key and the sign of the first value. Optionally, after determining the sign of the elements in the first key value, multiple mask tensors for the elements in the first key value can be constructed. The mask tensor of each element has the same dimension as the element, and the values in the mask tensor correspond one-to-one with the values in the element, representing the sign of the corresponding value in the element. The mask tensor of each element uses a binary representation to indicate the sign of the element; for example, 1 represents a negative value and 0 represents a positive value.
[0069] Taking an element as the first key as an example, the sign of the first key can be extracted. For instance, if the first key is positive, the corresponding value in the mask tensor is 0; if the first key is negative, the corresponding value in the mask tensor is 1. Based on multiple first keys, multiple values in the mask tensor of the first key are determined, constructing a mask tensor for the first key. The constructed mask tensor for the first key has the same dimension as the multiple first keys, meaning each first key corresponds to a value in its mask tensor. Similarly, based on multiple first values, multiple values in the mask tensor of the first value are determined, constructing a mask tensor for the first value. The constructed mask tensor for the first value has the same dimension as the multiple first values, meaning each first value corresponds to a value in its mask tensor.
[0070] Determine the absolute value of each element in the first key value by performing absolute value transformations on multiple first keys and multiple first values to obtain the absolute values of multiple first keys and multiple first values.
[0071] S302. Transform the absolute value of the element in the first key value into the corresponding second key value.
[0072] Transform the absolute value of the element in the first key value to the corresponding second key value. That is, transform the absolute value of the first key to obtain the corresponding second key, and transform the absolute value of the first value to obtain the corresponding second value.
[0073] The absolute value of each element in the first key value is transformed into the corresponding second key value. The specific execution method can be found in step S101, which describes how to convert each first key value in multiple first key values into the corresponding second key value.
[0074] S303. Quantize the second key value to obtain the corresponding third key value, and store the positive and negative signs of the elements in the third key value and the corresponding first key value.
[0075] If the mask tensor for the first key and the mask tensor for the first value were constructed in step S301, then the third key value, the mask tensor for the first key, and the mask tensor for the first value are stored. Each third key corresponds to a value in the mask tensor for the first key, and each third value corresponds to a value in the mask tensor for the first value.
[0076] S304. Dequantize the third bond value to obtain the dequantized third bond value.
[0077] The third bond and the third value are dequantized to obtain the dequantized third bond and the dequantized third value.
[0078] S305. Convert the dequantized third key value into the initial fourth key value.
[0079] The initial fourth key value includes the initial fourth key and the initial fourth value. Using the inverse operation method of step S302, the dequantized third key and the dequantized third value are converted respectively to obtain the initial fourth key and the initial fourth value. The initial fourth key and the initial fourth value are both numerical values without positive or negative signs.
[0080] S306. Add the sign of the element in the first key value corresponding to the third key value before the element corresponding to the initial fourth key value to obtain the fourth key value.
[0081] Add the sign of the first key corresponding to the third key before the corresponding initial fourth key to obtain the fourth key. Similarly, add the sign of the first value corresponding to the third value before the corresponding initial fourth value to obtain the fourth value.
[0082] For example, when determining the sign of an element in the first key value using the element's mask tensor, the fourth key value is obtained by adding the corresponding value from the mask tensor before the element of the initial fourth key value. Taking the constructed mask tensors for the first key and the first value as an example, based on the position of each initial fourth key within the fourth key, the value at the same position in the first key's mask tensor is obtained and added before the corresponding initial fourth key to obtain the fourth key. Similarly, based on the position of each initial fourth value within the fourth value, the value at the same position in the first value's mask tensor is obtained and added before the corresponding initial fourth value to obtain the fourth value.
[0083] The following specific embodiment illustrates the key-value processing method proposed in this application. Taking the square root operation method as an example, transforming the absolute value of the first key value into the corresponding second key value allows the data range of multiple second keys to be smaller than the data range of multiple first keys, the data range of multiple second values to be smaller than the data range of multiple first values, the outlier degree of outliers among multiple second keys to be less than the outlier degree of corresponding outliers among multiple first keys, and the outlier degree of outliers among multiple second values to be less than the outlier degree of corresponding outliers among multiple first values. Figure 4 A flowchart illustrating a key-value processing method proposed in this application. Figure 4 .
[0084] S401. Construct a mask tensor for the elements in the multiple first key values, and determine the absolute value of the elements in the first key values.
[0085] Taking the element mask tensor as an example, where 1 represents a negative value and 0 represents a positive value, we construct a mask tensor for each first key by determining multiple values in the mask tensor of the first key, and a mask tensor for each first value by determining multiple values in the mask tensor of the first value.
[0086] Determine the absolute value of the first key in the first key value, and the absolute value of the first value.
[0087] S402. Perform a square root operation on the absolute value of the element in the first key value to obtain the second key value.
[0088] The second key value is obtained by performing square root operations on the absolute value of the first key and the absolute value of the first key.
[0089] S403. Quantize the second key value to obtain the corresponding third key value, and store the positive and negative signs of the elements in the third key value and the corresponding first key value.
[0090] Each third key corresponds to a value in the mask tensor of the first key, and each third value corresponds to a value in the mask tensor of the first value.
[0091] S404. Dequantize the third bond value to obtain the dequantized third bond value.
[0092] S405. Squaring the elements in the dequantized third key value yields the initial fourth key value.
[0093] The squaring operation is the inverse operation of the square root operation in step S402. The square root operation is performed on the dequantized third key, and the dequantized third key, respectively, to obtain the initial fourth key value.
[0094] S406. Based on the mask tensor of the element, add the value at the corresponding position in the mask tensor before the element with the initial fourth key value to obtain the fourth key value.
[0095] Steps S401-S403 constitute the key-value storage stage. By performing a square root operation on the absolute value of each element of a first key-value pair, the corresponding second key-value pair is converted. This square root operation narrows the data range of the first keys and first values, and reduces the outlier nature of outliers within the first keys and first values. Since numerical quantization is performed on relatively concentrated data, the quantization error is relatively small. Therefore, numerical quantization based on the converted key-value pairs reduces the quantization error of the key-value cache and improves the accuracy of the inference results from the large language model.
[0096] Steps S404-S406 constitute the key-value recovery stage. The stored third key-value is subjected to squaring and dequantization to recover the key-value, resulting in the fourth key-value. This avoids redundant calculations and improves processing speed. Specifically, by quantizing the second key-value, the third key-value is obtained, saving storage space compared to storing the first key-value.
[0097] The key-value processing method in this application converts the absolute value of each element of a plurality of first key values into a corresponding second key value by performing a square root operation. This square root operation narrows the data range of the first keys and first values and reduces the outlier nature of outliers within the first keys and first values. Since numerical quantization is performed on relatively concentrated data, the quantization error is relatively small. Therefore, numerical quantization based on the converted key values reduces the quantization error of the key-value cache and improves the accuracy of inference results from large language models.
[0098] Optionally, based on the key-value processing method of this application, the effect of reducing quantization error can be further analyzed.
[0099] (1) Obtain the quantization error of the first key value based on the quantization method in the relevant technology. The quantization method in related technologies, for example, could be an asymmetric quantization method that converts a 16-bit floating-point quantization into an 8-bit integer, quantizing the first key and the first value in the first key value separately. Taking the first value as an example, the quantization error of the first value is obtained. .
[0100] Equation (1)
[0101] Where n represents the number of the first value, It is the i-th first value, which will Perform numerical quantization and storage, then process the quantized data. Perform inverse quantization to obtain .
[0102] (2) Obtain the key-value processing method based on the embodiments of this application, and process the quantization error of the first value. For example, using the embodiment exemplified by steps S401-S406, in step S403, the second key value is numerically quantized using the same quantization method as in step (1) to obtain the corresponding third key value.
[0103] Equation (2)
[0104] Where n represents the number of the first value, It is the i-th first value. Based on key-value processing method The corresponding i-th fourth key.
[0105] (3) Calculate the optimization of quantization error by using the key-value processing method in step (2) compared to the quantization method in step (1).
[0106] Equation (3)
[0107] (4) Acquiring spatial loss Space loss This represents the space loss caused by the additional storage space occupied by the key-value processing method in step (2) compared to the quantization method in step (1). Because the key-value processing method in step (2) introduces the storage of the positive and negative signs of the elements in the first key-value pair, compared to the quantization method in step (1), the additional space is the storage space occupied by storing the positive and negative signs of the elements in the first key-value pair. (Obtaining the space loss) The ratio of the additional storage space occupied to the storage space required by the quantization method in the relevant technology can be calculated. For example, in steps (1) and (2), the quantization adopts an asymmetric quantization method that quantizes a 16-bit floating point into an 8-bit integer, and in step (2), the positive and negative signs of the elements are represented by a binary method through a mask tensor, that is, the positive and negative signs of the first key are represented by 0 or 1, occupying 1 bit, so it is calculated by the following formula (4).
[0108] Equation (4)
[0109] (5) Obtain comprehensive and measurable benefits Calculate the quantization error optimization obtained through key-value processing, and the space loss caused by key-value processing. As shown in equation (5):
[0110] Equation (5)
[0111] Calculate the comprehensive measured benefit using equation (5) Furthermore, a revenue threshold can be preset; if the overall measured revenue is obtained... When the return exceeds a certain threshold, it indicates that the key-value processing method generates a return. For example, taking a preset return threshold of 1 as an example, if the obtained comprehensive measured return... If the value is greater than 1, it can be considered that, considering the optimization of the obtained quantization error and the resulting space loss, the key-value processing method has a comprehensive benefit.
[0112] This application also proposes a text processing method for obtaining text to be processed. The text to be processed is input into a pre-trained large language model to obtain the output response of the large language model. The large language model adopts a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts the key-value processing method as described in the embodiments of this application.
[0113] The key-value processing method of this application can reduce quantization error and improve the accuracy of text processing. Furthermore, after calculating the key value, the key value can be reused based on the key-value cache, which can improve the speed of text processing. Therefore, the text processing method of this application can improve the accuracy and speed of text processing.
[0114] Figure 5 This is a schematic diagram of a key-value processing device provided in an embodiment of this application. Figure 5 As shown, the key value processing device 500 includes a conversion module 501, a quantization module 502, and may also include a recovery module.
[0115] The conversion module 501 is used to convert each of the multiple first key values into a corresponding second key value during the key value calculation process of the self-attention mechanism of the large language model. The first key value includes a first key and a first value, and the second key value includes a second key and a second value. The data range of the multiple second keys is smaller than the data range of the multiple first keys, and the data range of the multiple second values is smaller than the data range of the multiple first values. The outlier degree of the outlier values among the multiple second keys is smaller than the outlier degree of the corresponding outlier values among the multiple first keys, and the outlier degree of the outlier values among the multiple second values is smaller than the outlier degree of the corresponding outlier values among the multiple first values.
[0116] The quantization module 502 is used to quantize the second key value to obtain the corresponding third key value and store the third key value.
[0117] In one implementation, the recovery module is specifically used to dequantize the third key value to obtain the dequantized third key value; and to convert the dequantized third key value into the corresponding fourth key value; wherein, the operation of converting the dequantized third key value into the corresponding fourth key value is the inverse operation of converting each of the multiple first key values into the corresponding second key value.
[0118] In one implementation, the conversion module 501 is specifically used to determine the sign of the element in the first key value and to determine the absolute value of the element in the first key value, wherein the element is the first key or the first value; and to transform the absolute value of the element in the first key value into the corresponding second key value.
[0119] Accordingly, in this implementation, the quantization module 503 is specifically used to store the positive and negative signs of the elements in the third key value and the corresponding first key value.
[0120] In one implementation, the recovery module is specifically used to convert the dequantized third key value into an initial fourth key value; the positive or negative sign of the element in the first key value corresponding to the third key value is added before the corresponding element in the initial fourth key value to obtain the fourth key value.
[0121] In one implementation, after determining the sign of the elements in the first key value, the conversion module 501 is specifically used to construct a mask tensor of multiple elements in the first key value; wherein, the mask tensor of the elements has the same dimension as the elements, and the mask tensor of the elements uses a binary method to represent the sign of the elements;
[0122] Accordingly, the recovery module is specifically used to convert the dequantized third key value into the initial fourth key value; based on the mask tensor of the element, the value at the corresponding position in the mask tensor is added before the element of the initial fourth key value to obtain the fourth key value.
[0123] In one implementation, the conversion module 501 is specifically used to perform a square root operation on the absolute value of the elements in the first key value to obtain the second key value; the recovery module performs a square operation on the elements in the dequantized third key value to obtain the initial fourth key value.
[0124] Figure 6 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application. Figure 6 As shown, the text processing device 600 includes:
[0125] Module 601 is used to acquire the text to be processed;
[0126] The processing module 602 is used to input the text to be processed into a pre-trained large language model and obtain the response output by the large language model. The large language model adopts a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts any of the key-value processing methods in the above embodiments.
[0127] Figure 7 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 may include at least one processor 701 for implementing the system fault prediction method provided in the embodiments of this application.
[0128] Optionally, the electronic device 700 further includes at least one memory 702 for storing program instructions and / or data. The memory 702 is coupled to the processor 701. The coupling in this embodiment is an indirect coupling or communication connection between devices, units, or modules, and can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 701 may operate in conjunction with the memory 702. The processor 701 may execute program instructions stored in the memory 702. At least one of the at least one memory may be included in the processor.
[0129] Optionally, the electronic device 700 further includes a communication interface 703 for communicating with other devices via a transmission medium, thereby enabling the electronic device 700 to communicate with other devices. The communication interface 703 may be, for example, a transceiver, interface, bus, circuit, or a device capable of transmitting and receiving functions. The processor 701 can utilize the communication interface 703 to transmit and receive data and / or information, and to implement the methods provided in the embodiments of this application. For details, please refer to the detailed descriptions in the preceding embodiments; further elaboration is not repeated here.
[0130] This application embodiment does not limit the specific connection medium between the processor 701, memory 702, and communication interface 703. This application embodiment... Figure 7 The processor 701, memory 702, and communication interface 703 are connected via bus 704. Bus 704 is... Figure 7 The connections between other components are shown in thick lines only and are not intended to be limiting. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0131] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.
[0132] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0133] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run by a processor, it causes the computer to perform the methods as described in any of the foregoing embodiments.
[0134] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the foregoing embodiments.
[0135] The terms “unit”, “module”, etc., used in this specification may be used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution.
[0136] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0137] The unit described as a separate component may or may not be physically separate. The component shown as a unit may or may not be a physical unit; that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0138] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0139] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., digital video disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).
[0140] If this function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0141] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0142] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A key-value processing method, characterized by, The method comprises: In the key-value calculation process of the self-attention mechanism of the large language model, each of a plurality of first key-values is respectively converted into a corresponding second key-value, wherein the first key-value comprises a first key and a first value, the second key-value comprises a second key and a second value, the data range of the plurality of second keys is smaller than the data range of the plurality of first keys, and the data range of the plurality of second values is smaller than the data range of the plurality of first values; the outlying degree of an outlying value in the plurality of second keys is smaller than the outlying degree of a corresponding outlying value in the plurality of first keys, and the outlying degree of an outlying value in the plurality of second values is smaller than the outlying degree of a corresponding outlying value in the plurality of first values; Numerical quantization is performed on the second key-value to obtain a corresponding third key-value, and the third key-value is stored.
2. The method of claim 1, wherein, Further comprising: Dequantizing the third key-value to obtain a dequantized third key-value; The dequantized third key-value is converted into a corresponding fourth key-value; wherein the operation of converting the dequantized third key-value into the corresponding fourth key-value is the inverse operation of the operation of converting each of the plurality of first key-values into the corresponding second key-value.
3. The method of claim 2, wherein, The operation of converting each of the plurality of first key-values into the corresponding second key-value comprises: Determining the positive and negative signs of elements in the first key-value, and determining the absolute values of the elements in the first key-value, the elements being the first key or the first value; Transforming the absolute values of the elements in the first key-value into corresponding second key-values; The storage of the third key-value comprises: The third key-value and the positive and negative signs of the elements in the corresponding first key-value are stored correspondingly.
4. The method of claim 3, wherein, The operation of converting the dequantized third key-value into the corresponding fourth key-value comprises: The dequantized third key-value is converted into an initial fourth key-value; The positive and negative signs of the elements in the first key-value corresponding to the third key-value are added in front of the corresponding elements in the initial fourth key-value to obtain the fourth key-value.
5. The method of claim 3, wherein, After determining the positive and negative signs of the elements in the first key-value, the method further comprises: A mask tensor of the elements in the plurality of first key-values is constructed; wherein the mask tensor of the elements is the same as the dimension of the elements, and the mask tensor of the elements represents the positive and negative signs of the elements by using a binary method; The operation of converting the dequantized third key-value into the corresponding fourth key-value comprises: The dequantized third key-value is converted into an initial fourth key-value; According to the mask tensor of the elements, a numerical value at a corresponding position in the mask tensor is added in front of an element in the initial fourth key-value to obtain the fourth key-value.
6. The method of claim 4, wherein, The operation of transforming the absolute values of the elements in the first key-value into corresponding second key-values comprises: Square root operation is performed on the absolute values of the elements in the first key-value to obtain the second key-value; The operation of converting the dequantized third key-value into an initial fourth key-value comprises: Square operation is performed on the elements in the dequantized third key-value to obtain the initial fourth key-value.
7. A text processing method characterized by, The method comprises: Obtaining a to-be-processed text; The text to be processed is input into a pre-trained large language model to obtain the response output by the large language model. The large language model adopts a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts the key-value processing method as described in any one of claims 1-6.
8. A key value processing apparatus characterized by comprising: include: The conversion module is used to convert each of a plurality of first key values into a corresponding second key value during the key value calculation process of the self-attention mechanism of the large language model. The first key value includes a first key and a first value, and the second key value includes a second key and a second value. The data range of the plurality of second keys is smaller than the data range of the plurality of first keys, and the data range of the plurality of second values is smaller than the data range of the plurality of first values. The outlier degree of the outlier values among the plurality of second keys is smaller than the outlier degree of the corresponding outlier values among the plurality of first keys, and the outlier degree of the outlier values among the plurality of second values is smaller than the outlier degree of the corresponding outlier values among the plurality of first values. The quantization module is used to perform numerical quantization on the second key value to obtain the corresponding third key value, and to store the third key value.
9. A text processing apparatus characterized by comprising: include: The acquisition module is used to acquire the text to be processed; The processing module is used to input the text to be processed into a pre-trained large language model and obtain the response output by the large language model. The large language model adopts a self-attention mechanism, and the key-value calculation process of the self-attention mechanism of the large language model adopts the key-value processing method as described in any one of claims 1-6.
10. An electronic device, comprising: include: Memory and processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory, wherein the computer program, when executed, causes the processor to perform the method of any one of claims 1-7.
11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1-7.
12. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.