Method, apparatus, computing device cluster, and storage medium for key-value quantization

Abstract

The application discloses a key-value quantization method and device, a computing device cluster and a storage medium. The method comprises the following steps: obtaining a first quantization strategy corresponding to any attention layer of a neural network model; the first quantization strategy comprises a measurement mode of a key K vector and a value V vector output by the any attention layer; determining a key token in a first token sequence obtained based on user input based on at least one of the K vector and the V vector output by the any attention layer and the measurement mode of the at least one vector; determining a first target token in the first token sequence which needs to be quantized in the K vector and the V vector based on the key token; and quantizing the K vector and the V vector corresponding to the first target token. The key token identification is beneficial to improving the accuracy of KV cache quantization.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus, computing device cluster and storage medium for key-value quantization. Background Technology

[0002] Large Language Models (LLMs) are deep learning models pre-trained on massive amounts of data. They can not only generate natural language text but also deeply understand its meaning, handling various natural language tasks such as text summarization, question answering, and translation. LLMs require efficient storage and computation during inference. Key-value (K) caching can reduce the complexity of LLM inference decoding and is an effective solution for accelerating LLM inference. However, with the increase in input sequence size, batch size, and model parameters, the size of the KV cache also increases, and the storage space occupied by the KV cache has become a bottleneck limiting model throughput. To address the KV cache problem, KV cache compression technology has gradually become a hot research topic in the field, but the compression accuracy of existing KV compression techniques still needs improvement. Summary of the Invention

[0003] This application provides a method, apparatus, computing device cluster, and storage medium for key-value quantization. By applying different quantization strategies to the K-vectors and V-vectors output by each attention layer of the neural network model, and by detecting and identifying key tokens based on the quantization strategies of each attention layer, the full-precision representation of key tokens is preserved. Quantization of K-vectors and V-vectors is performed on tokens other than key tokens, which helps to improve the accuracy of KV cache quantization.

[0004] In a first aspect, embodiments of this application provide a key-value quantization method, applied to an electronic device or a chip in an electronic device. Taking the application of this method to an electronic device as an example, the method includes: receiving user input; obtaining a first token sequence based on the user input, the first token sequence including multiple tokens; for an attention layer of a neural network model, generating a key K vector and a value V vector corresponding to each of the multiple tokens based on the first token sequence; obtaining a first quantization strategy corresponding to the attention layer, the first quantization strategy including a measurement method for at least one of the K vectors and V vectors generated by the attention layer; determining a first target K vector and a first target V vector to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer based on the at least one vector generated by the attention layer and the measurement method of the at least one vector; and quantizing the first target K vector and the first target V vector.

[0005] As can be seen in this embodiment, for any attention layer, the electronic device can obtain the quantization strategy (i.e., the first quantization strategy) corresponding to that attention layer, and identify the first target K vector and the first target V vector that need to be quantized from the generated K vector and V vector based on at least one of the K vectors and V vectors output by the attention layer and the measurement method of the at least one vector in the quantization strategy, and then quantize them. For the first target K vector and the first target V vector, the quantization parameters in the quantization strategy are used to quantize them, and the introduced quantization error is usually small, which is beneficial to improving the accuracy of KV buffer quantization. As for the K vectors and V vectors other than the first target K vector and the first target V vector in the output K vector and V vector, they are generally considered to have a more abnormal distribution and contain more special information, so their full-precision representation is retained during KV quantization. Furthermore, since the quantization strategies corresponding to each attention layer are different, electronic devices can achieve a balance between compression ratio and compression effect through the conversion of inter-layer quantization strategies during the quantization process. For example, in some attention layers, the feature information may not be abundant (or the feature distribution is relatively uniform), so a high compression ratio can be used to reduce the amount of computation and memory usage. In other attention layers, the feature information may be important (or the feature distribution is relatively abnormal), so a low compression ratio is needed to improve the compression effect. The dynamic balance between compression ratio and compression effect is beneficial to improving the memory utilization of computing chips, increasing the throughput of model inference, and reducing inference latency.

[0006] In one possible implementation, based on at least one vector generated by the attention layer and at least one vector metric method, the first target K vector and the first target V vector that need to be quantized among multiple K vectors and multiple V vectors generated by the attention layer are determined, including: based on at least one vector generated by the attention layer and at least one vector metric method, determining a key token among multiple tokens, determining a first target token that needs to be quantized by K vector and V vector in the first token sequence based on the key token, and determining the K vector and V vector corresponding to the first target token among multiple K vectors and multiple V vectors as the first target K vector and the first target V vector.

[0007] In this implementation, based on the K-vector and / or V-vector metric methods, the electronic device can identify key tokens with abnormal feature distributions in the token sequence (i.e., the first token sequence) obtained based on user input. Since the information contained in the key tokens is quite specific, quantizing their K-vectors and V-vectors would lead to significant quantization errors. Therefore, during KV quantization, the full-precision representation of the key tokens is retained. All tokens in the first token sequence other than the key tokens are identified as the first target tokens. The K-vectors and V-vectors corresponding to the first target tokens are the first target K-vectors and first target V-vectors that need to be quantized.

[0008] In one possible implementation, the metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold. The metric value corresponding to each token is determined based on at least one of the K vector and V vector corresponding to each token and the metric method of at least one vector.

[0009] In this implementation, tokens in the first token sequence whose metric value is greater than or equal to a first threshold, or whose metric value is less than or equal to a second threshold, are identified as key tokens. The distributions of the K-vectors and / or V-vectors of these key tokens are relatively anomalous. Quantizing their K-vectors and V-vectors may introduce significant quantization errors. Therefore, dynamically identifying these key tokens helps preserve their full-precision vector representation, thereby reducing quantization errors.

[0010] In one possible implementation, the metric values ​​corresponding to the first target K vector and the first target V vector are less than a third threshold, or the metric values ​​corresponding to the first target K vector and the first target V vector are greater than a fourth threshold. The metric values ​​corresponding to the multiple K vectors and the multiple V vectors are determined based on at least one vector and at least one vector measurement method.

[0011] In this implementation, the metric values ​​corresponding to multiple K vectors and multiple V vectors actually refer to the metric values ​​corresponding to multiple tokens. The electronic device can determine the metric value corresponding to each token based on at least one of the K vectors and V vectors corresponding to each token and the metric method of the at least one vector. The K vectors and V vectors corresponding to tokens whose metric values ​​are less than the third threshold or greater than the fourth threshold are determined as the first target K vector and the first target V vector, respectively. Therefore, it is not necessary to determine the first target K vector and the first target V vector based on the key token, which is beneficial to improving the recognition efficiency of the first target K vector and the first target V vector.

[0012] In one possible implementation, the first quantization strategy further includes calculating the number N of historical nearest neighbor tokens on which the attention value of each token depends, and determining the first target token in the first token sequence that needs to be quantized by K-vector and V-vector based on the key token. This includes: for the i-th token in the first token sequence that needs to have its attention value calculated at the current time step, determining the historical nearest neighbor token sequence on which the attention value of the i-th token depends based on N, and determining the tokens other than the key token among the tokens before the historical nearest neighbor token sequence as the target tokens that need to be quantized by K-vector and V-vector before the i-th token. The first target tokens include the target tokens that need to be quantized by K-vector and V-vector before the i-th token.

[0013] In this implementation, the electronic device can perform a target token lookup operation when calculating the attention of each token. Since the first quantization strategy includes the number N of historical nearest neighbor tokens on which the attention value of each token depends, the electronic device needs to first determine the sequence of historical nearest neighbor tokens on which the i-th token depends when determining the target token. Then, it needs to determine the target token that needs to be quantized by K vector and V vector from the tokens before the historical nearest neighbor token sequence, so as to avoid the K vector and V vector of the historical nearest neighbor tokens on which the i-th token depends being quantized, thereby affecting the accuracy of the attention calculation of the i-th token.

[0014] In one possible implementation, the first quantization strategy also includes the number of tokens M required to perform one quantization, and quantizing the first target K vector and the first target V vector, including: when the number of target tokens Q that need to be quantized before the i-th token is greater than or equal to M, quantizing the K vector and V vector corresponding to the Q target tokens or M target tokens among the Q target tokens.

[0015] In this implementation, the first quantization strategy defines the number M of tokens required to perform one quantization. For the i-th token used to calculate attention at the current time step, if the number of target tokens requiring K-vector and V-vector quantization before this token is greater than or equal to M, the electronic device is triggered to perform one K-vector and V-vector quantization. Specifically, for the K-vectors and V-vectors corresponding to Q target tokens or M of the Q target tokens, the electronic device can perform quantization along the channel dimension or along the token dimension, offering relatively flexible quantization methods.

[0016] In one possible implementation, the first quantization strategy further includes the number P of key tokens preset in any attention layer, where P is less than the length of the first token sequence. The method further includes: if the number of key tokens determined in the first token sequence is equal to P, and if a first key token is further determined in the first token sequence, then the first key token is used to replace the second key token among the P key tokens that have been determined. The difference between the metric value corresponding to the first key token and the judgment threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the judgment threshold used to determine the second key token.

[0017] In this implementation, for any attention layer, the number P of key tokens is limited to be less than the length of the first token sequence. This prevents the entire first token sequence from being identified as key tokens, ensuring that non-key tokens are quantized in the current attention layer. If, when P key tokens are identified, another key token is identified in the first token sequence, and the distribution of its K and / or V vectors is more anomalous than the distribution of any of the existing P key tokens, the electronic device replaces the less anomalous key token with the newly identified key token. This ensures that the more anomalous key token is detected and retains the full-precision representation of such tokens, thereby reducing quantization errors.

[0018] In one possible implementation, the method further includes providing a first user interface for displaying the key token.

[0019] In this implementation, the electronic device can output the key tokens identified at any attention layer. Because different attention layers employ different quantization strategies, especially regarding the number and measurement methods of key tokens, the key tokens identified by different attention layers will also differ. Based on the output of the first-user interface, the user can intuitively perceive that the identified key tokens are dynamically changing, rather than fixed.

[0020] In one possible implementation, the method further includes: receiving a first user instruction for adjusting the metric of at least one vector in a first quantization strategy.

[0021] In this implementation, since the key tokens identified by each attention layer can be visualized, if the output key tokens do not meet the requirements, the electronic device can adjust the measurement method of at least one vector in the first quantization strategy based on the first user instruction (for example, the first user instruction includes at least one new measurement method for the vector) to further optimize the quantization strategy at the layer granularity, thereby improving the accuracy of key token identification in each attention layer, and further improving the quantization accuracy of the K vector and V vector of the target token in each attention layer.

[0022] In one possible implementation, the method further includes: receiving a second user instruction for adjusting parameter P in the first quantization strategy.

[0023] In this implementation, since the key tokens identified by each attention layer can be visualized, if the output key tokens do not meet the requirements, the electronic device can adjust the parameter P in the first quantization strategy based on the second user instruction (for example, the first user instruction includes a new P value) to further optimize the quantization strategy at the layer granularity. This is beneficial to improving the accuracy of key token identification in each attention layer, and in turn, to improving the quantization accuracy of the K vector and V vector of the target token in each attention layer.

[0024] In one possible implementation, the neural network model includes multiple attention layers; before receiving user input and obtaining the first token sequence based on the user input, the method further includes: during the process of reasoning on sample data using the neural network model, for any attention layer, obtaining the second quantization strategy corresponding to any attention layer, quantizing the K vector and V vector corresponding to the second target token based on the second quantization strategy, obtaining the KV cache quantized by any attention layer, wherein the second target token is determined based on the key tokens in the second token sequence, the second token sequence is obtained based on the sample data, and based on the KV cache quantized by at least one attention layer, determining one or more performance indicators for the neural network model to reason on the sample data, providing a second user interface for displaying one or more performance indicators, and receiving a third user instruction for adjusting the second quantization strategy to obtain the first quantization strategy.

[0025] In this implementation, after deploying the neural network model, the user can use sample data as input to test one or more performance metrics of the neural network model's inference. During the inference process on the sample data, the electronic device identifies key tokens in the second token sequence obtained based on the sample data using a second quantization strategy corresponding to any attention layer, and quantizes the K-vector and V-vector corresponding to the second target token in the second token sequence. Based on the quantization results of at least one attention layer, the electronic device can determine one or more performance metrics of the neural network model's inference on the sample data and display one or more performance metrics through a second user interface. If one or more performance metrics do not meet the user's requirements, the electronic device can receive a third user instruction and adjust all or part of the quantization parameters in the second quantization strategy based on the third user instruction (e.g., the third user instruction includes the updated value of the quantization parameters to be adjusted) to obtain a first quantization strategy that meets different performance metric requirements. This facilitates the use of a first quantization strategy that meets different performance metric requirements for KV cache quantization in later application stages, thereby achieving a balance between the performance metrics of the neural network model's inference and compression requirements, and realizing finer-grained KV cache quantization.

[0026] In one possible implementation, a third user instruction is used to indicate the metric of at least one vector generated by the attention layer, wherein the metric of at least one vector includes one or more of the following: L1 norm, L2 norm, L∞ norm, and Lp norm.

[0027] In this implementation, the user can instruct the electronic device via a third user instruction to specify the metric of at least one of the K-vectors and V-vectors generated by any attention layer. For example, the metric can be one or more of the L1 norm, L2 norm, L∞ norm, and Lp norm, so that the electronic device can subsequently use the metric of at least one of the K-vectors and V-vectors to identify the key tokens in the first token sequence.

[0028] In one possible implementation, before reasoning about the sample data using a neural network model, the method further includes: displaying a third user interface for any attention layer, the third user interface being an interface for setting quantization parameters for the K vector and V vector output by any attention layer, the third user interface including one or more setting areas among at least one vector measurement setting area, N setting area, M setting area and P setting area, and obtaining a second quantization strategy based on the user's operation on one or more setting areas.

[0029] In this implementation, for any attention layer of the neural network model, a quantization parameter setting interface (i.e., a third user interface) for K vectors and V vectors is provided. In addition to the setting area for conventional quantization parameters such as quantization bits, this interface also includes one or more setting areas for the measurement method of at least one of the K vectors and V vectors, the setting area for N, the setting area for M, and the setting area for P. During the deployment phase of the neural network model, based on the user's operation on the one or more setting areas, the electronic device can generate an initial quantization strategy (i.e., a second quantization strategy) for any attention layer, thereby realizing the visualization of quantization parameters.

[0030] Secondly, embodiments of this application provide a key-value quantization apparatus, applied to an electronic device or a chip in an electronic device; the apparatus includes an acquisition unit and a processing unit; wherein: the acquisition unit is used to receive user input and obtain a first token sequence based on the user input, the first token sequence including multiple tokens; the processing unit is used to generate a key K vector and a value V vector corresponding to each of the multiple tokens based on the first token sequence for an attention layer of a neural network model; the acquisition unit is also used to acquire a first quantization strategy corresponding to the attention layer, the first quantization strategy including a measurement method for at least one of the K vectors and V vectors generated by the attention layer; the processing unit is also used to determine a first target K vector and a first target V vector that need to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer based on the at least one vector generated by the attention layer and the measurement method of the at least one vector; and to quantize the first target K vector and the first target V vector.

[0031] In one possible implementation, regarding determining the first target K vector and the first target V vector that need to be quantized among multiple K vectors and multiple V vectors generated by the attention layer based on at least one vector generated by the attention layer and at least one vector metric method, the processing unit is specifically used to: determine the key token among multiple tokens based on at least one vector generated by the attention layer and at least one vector metric method; determine the first target token that needs to be quantized by K vector and V vector in the first token sequence based on the key token; and determine the K vector and V vector corresponding to the first target token among multiple K vectors and multiple V vectors as the first target K vector and the first target V vector.

[0032] In one possible implementation, the metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold. The metric value corresponding to each token is determined based on at least one of the K vector and V vector corresponding to each token and the metric method of at least one vector.

[0033] In one possible implementation, the metric values ​​corresponding to the first target K vector and the first target V vector are less than a third threshold, or the metric values ​​corresponding to the first target K vector and the first target V vector are greater than a fourth threshold. The metric values ​​corresponding to the multiple K vectors and the multiple V vectors are determined based on at least one vector and at least one vector measurement method.

[0034] In one possible implementation, the first quantization strategy further includes calculating the number N of historical nearest neighbor tokens on which the attention value of each token depends. In determining the first target token in the first token sequence that needs to be quantized by K-vector and V-vector based on the key token, the processing unit is specifically used to: for the i-th token in the first token sequence that needs to have its attention value calculated at the current time step, determine the historical nearest neighbor token sequence on which the attention value of the i-th token depends based on N, and determine the tokens other than the key token among the tokens before the historical nearest neighbor token sequence as the target tokens that need to be quantized by K-vector and V-vector before the i-th token. The first target tokens include the target tokens that need to be quantized by K-vector and V-vector before the i-th token.

[0035] In one possible implementation, the first quantization strategy also includes the number of tokens M required to perform one quantization. In terms of quantizing the first target K vector and the first target V vector, the processing unit is specifically used to: when the number of target tokens Q that need to be quantized before the i-th token is greater than or equal to M, quantize the K vector and V vector corresponding to the Q target tokens or M target tokens among the Q target tokens.

[0036] In one possible implementation, the first quantization strategy further includes a number P of key tokens preset by the attention layer, where P is less than the length of the first token sequence. The processing unit is further configured to: if the number of key tokens determined in the first token sequence is equal to P, and if a first key token is further determined in the first token sequence, then the first key token is used to replace the second key token among the P key tokens that have been determined. The difference between the metric value corresponding to the first key token and the judgment threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the judgment threshold used to determine the second key token.

[0037] In one possible implementation, the processing unit is also used to: provide a first user interface for displaying the key token.

[0038] In one possible implementation, the processing unit is further configured to: receive a first user instruction, the first user instruction being used to adjust the metric mode and / or P of at least one vector in the first quantization strategy.

[0039] In one possible implementation, the processing unit is further configured to: receive a second user instruction, which is used to adjust P in the first quantization strategy.

[0040] In one possible implementation, the neural network model includes multiple attention layers. The acquisition unit is further configured to: during the inference process of using the neural network model to infer sample data, for any attention layer, acquire the second quantization strategy corresponding to any attention layer. The processing unit is further configured to: quantize the K vector and V vector corresponding to the second target token based on the second quantization strategy to obtain the KV cache quantized by any attention layer, wherein the second target token is determined based on the key tokens in the second token sequence, the second token sequence is obtained based on the sample data, and based on the KV cache quantized by at least one attention layer, determine one or more performance indicators for the neural network model to infer sample data, provide a second user interface for displaying one or more performance indicators, and receive third user instructions for adjusting the second quantization strategy to obtain a first quantization strategy.

[0041] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0042] Thirdly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the embodiments of the first aspect above.

[0043] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any of the embodiments of the first aspect above.

[0044] Fifthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, implement the method as described in any of the embodiments of the first aspect above.

[0045] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0047] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0048] Figure 2 A schematic diagram illustrating quantization processing and attention calculation provided in an embodiment of this application;

[0049] Figure 3 A flowchart illustrating a key-value quantization method provided in an embodiment of this application;

[0050] Figure 4 A schematic diagram of a human-computer interaction interface provided in an embodiment of this application;

[0051] Figure 5 A schematic diagram illustrating the exclusion of a key token provided in an embodiment of this application;

[0052] Figure 6 A schematic diagram of another human-computer interaction interface provided in an embodiment of this application;

[0053] Figure 7 A schematic diagram illustrating memory usage during model inference, provided as an embodiment of this application;

[0054] Figure 8 A flowchart illustrating another key-value quantization method provided in this application embodiment;

[0055] Figure 9A A schematic diagram of a basic settings interface provided in an embodiment of this application;

[0056] Figure 9B This application provides a schematic diagram illustrating a successful model deployment.

[0057] Figure 10 A schematic diagram of a key-value quantization device provided in an embodiment of this application;

[0058] Figure 11 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0059] Figure 12A schematic diagram of a computing device cluster provided in an embodiment of this application;

[0060] Figure 13 This is a schematic diagram illustrating a connection method between computing devices provided in an embodiment of this application. Detailed Implementation

[0061] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0062] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0064] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0065] (1) Large Language Model (LLM): refers to a deep learning model with a large number of parameters that is pre-trained using a large amount of text data. These models can understand and generate natural language text and are widely used in natural language processing tasks such as text generation, translation, and question answering. These models usually have billions or more parameters and learn from a large amount of data to predict the next word or generate related content, playing an important role in the field of artificial intelligence.

[0066] (2) LLM Inference System: This refers to a system that provides inference services using a pre-trained LLM. The system receives text input from users or other systems, processes it through the LLM, and generates and returns the corresponding text output. It typically includes components such as an input processor, scheduler, large model engine, and return processor.

[0067] (3) LLM inference acceleration technology: including model compression, KV cache compression, fast decoding algorithm, low-level optimization and distributed parallel inference, etc., aiming to improve the inference speed and efficiency of LLM.

[0068] (4) Key-Value Cache Compression: This refers to the compression techniques used in key-value caches to improve cache efficiency and capacity by reducing data storage space. In actual deployments, compressing the key-value cache can reduce memory usage and increase batch size, thereby increasing parallelism and throughput.

[0069] (5) Quantization: A method to reduce the storage space required for data. This involves reducing the bit width of the data representation to decrease storage and computational requirements while maintaining the original data precision as much as possible. In this application, it refers to KV cache quantization, which means reducing the KV cache storage space while minimizing its computational errors.

[0070] (6) Token: The basic unit of input data in natural language processing. It can be a word, a subword, or even a character, depending on the tokenization strategy used. In some descriptions, a token can be a symbol, lexical unit, mark, lexical unit, etc.

[0071] (7) Throughput: A metric used to evaluate the inference speed of a system. In an LLM inference system, throughput is calculated as the number of output tokens divided by the output time, typically expressed as tokens per second. Higher throughput indicates faster inference speed and thus higher system efficiency.

[0072] (8) Memory: This usually refers to the memory in a computer or other device used to store data and programs. In LLM applications, memory can refer to the memory space used by the model during runtime, including but not limited to model parameters, intermediate calculation results, and input / output data.

[0073] KIVI is an asymmetric key-value (KV) cache quantization solution that significantly reduces memory consumption and improves processing speed by compressing the K cache by channel and the V cache by token to 2-bit precision. KIVI's basic assumption is that some channels of K have large amplitudes, and the distribution of K within the same channel is relatively uniform; therefore, quantizing K by channel can reduce errors. For V, since its distribution is token-dependent, it is quantized by token. KIVI groups the K quantization process, performing quantization once for each group of K and calculating the quantization coefficients, making the quantization results more closely resemble the actual data distribution. This method requires no additional training and effectively preserves model accuracy while compressing the KV cache size.

[0074] However, KIVI assumes a uniform channel distribution for K, but in real-world applications, some tokens exhibit anomalous KV distributions; these tokens are called key tokens. Because the KV amplitudes of these tokens deviate significantly from those of other tokens, the KIVI method leads to large quantization errors during quantization. Furthermore, these key tokens typically have high attention scores, and their quantization errors significantly impact the final inference result, reducing the model's inference accuracy. KIVI employs a uniform quantization strategy across all attention layers without differentiating attention mechanisms for different layers. In practice, the attention weights of the first few layers differ considerably from those of other layers, and the key values ​​in the first few layers are often more irregular, deviating from the channel-based distribution rule. KIVI fails to adjust for these differences, resulting in poor quantization performance in some layers and further limiting the overall quantization accuracy and performance of the model.

[0075] KVQuant is a quantization scheme that embeds multiple key-value (KV) caching compression strategies, including asymmetric quantization, non-uniform quantization, pre-rotational position embedding (Pre-RoPE) quantization, and outlier-aware quantization. Through training, KVQuant can better integrate the quantization and dequantization processes into the decoding process without requiring group-based quantization.

[0076] However, the training introduced by KVQuant incurs additional training overhead, and its generalization ability is difficult to verify. The asymmetric quantization coefficients generated during training are inferior to those calculated during actual inference. While this method simplifies the quantization process (eliminating the need for group accumulation), it introduces significant errors. Furthermore, although KVQuant considers some important tokens, the positions of these important tokens are usually fixed. Tokens not in these positions are still quantized, resulting in the loss of crucial information. Pre-RoPE quantization recalculates RoPE after dequantization during actual inference, adding extra computational cost. Moreover, the computational cost of RoPE is relatively high, impacting the model's throughput.

[0077] To overcome the defects and shortcomings of the prior art, embodiments of this application provide a key-value quantization method, which can be applied to... Figure 1 The system architecture is shown below. Figure 1 As shown, the system architecture provided in this application embodiment includes an input module, an attention module, an output module, and a monitoring and optimization module. The attention module includes a KV caching module, a quantization processing module, and an attention calculation module.

[0078] The input module is used to receive user input for inference of a neural network model (e.g., LLM). The user input may be a text sequence or other data format. The input module converts the user input into an input representation suitable for the Transformer architecture. The input representation is usually a sequence of tokens. The input module is also used to generate a corresponding embedding vector for each token.

[0079] The quantization module retrieves system-generated K-vectors and V-vectors from the KV cache module during inference. For each attention layer, it identifies key tokens based on predefined metric methods for K-vectors and / or V-vectors. Based on the identified key tokens, it determines the target tokens for K-vector and V-vector quantization in the current attention layer. For these target tokens, K-vector quantization uses a channel-level quantization strategy, and V-vector quantization uses a token-level quantization strategy; or K-vector quantization uses a token-level quantization strategy, and V-vector quantization uses a channel-level quantization strategy; or both K-vector and V-vector quantization use a channel-level quantization strategy; or both K-vector and V-vector quantization use a token-level quantization strategy.

[0080] The KV caching module caches the K and V vectors generated by the system, as well as the quantized K and V vectors output by the quantization processing module. The quantized K and V vectors will overwrite their corresponding full-precision K and V vectors in the KV caching module. If the user selects full-precision inference for a certain attention layer, the KV caching module will cache the full-precision K and V vectors for that layer.

[0081] The attention calculation module is used to calculate the attention value of the token based on the query (Q) vector and the quantized K and V vectors in the KV caching module.

[0082] For example, such as Figure 2 As shown, the quantization processing module may include a key token identification unit and a quantization unit. The system generates K and V vectors to create a KV cache. The key token identification unit retrieves the K and V vectors from the KV cache and performs key token identification based on the metric methods corresponding to the K and V vectors, respectively. Based on the key tokens, the target token in the token sequence is determined. The quantization unit quantizes the K and V vectors corresponding to the target token. The quantized K vector overwrites the corresponding full-precision K vector in the KV cache, and the quantized V vector overwrites the corresponding full-precision V vector in the KV cache. The attention calculation module retrieves the full-precision K and V vectors corresponding to the key token and the quantized K and V vectors corresponding to the target token from the KV cache, and performs attention calculation in conjunction with the Q vector.

[0083] The output module generates the in-layer results for each attention layer through processing by subsequent network layers (such as feedforward layers, fully connected layers, etc.). Furthermore, after processing by multiple attention layers and a decoding head with the same structure, it generates the model inference result corresponding to the user input. For example, this inference result can be the probability distribution of a token, which is then sampled or processed to obtain the final output text sequence or other formatted prediction results.

[0084] The monitoring and optimization module is used to monitor system performance metrics in real time, including but not limited to memory usage, high-bandwidth memory (HBM) usage, compute card utilization, model throughput, model accuracy loss, compression ratio, etc., providing feedback to optimize KV cache management strategies and quantization parameters. Furthermore, it can monitor the quantization parameters of each layer, the KV cache space occupied by each layer, and the key tokens identified at each layer. Users can adjust the quantization parameters of each layer based on the monitoring results.

[0085] It should be understood that in the above system architecture, different quantization strategies are used for each attention layer of the model. For example, the K-vectors and V-vectors output by different attention layers are measured in different ways. Therefore, the key tokens output by the monitoring and optimization module for each attention layer will be different, and the number of key tokens will also be different. Through the collaborative work of the core components in the above system architecture, this application can achieve efficient KV cache quantization management, effectively reduce memory usage with low precision loss, and help improve the inference speed and throughput of LLM in long text processing, recommendation systems, and customer service systems.

[0086] For example, Figure 1 The system architecture shown can be applied to electronic devices, such as personal computers (PCs), desktops, all-in-ones, servers, cloud devices, workstations, etc.

[0087] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0088] Please see Figure 3 , Figure 3 This is a flowchart illustrating a key-value quantization method provided in an embodiment of this application, which can be applied to electronic devices. Figure 3 As shown, the method includes steps 301-305:

[0089] 301: Receive user input and obtain the first token sequence based on the user input. The first token sequence includes multiple tokens.

[0090] For the user input to be processed, the electronic device converts it into a token sequence (i.e., the first token sequence mentioned below) through the input embedding layer of the neural network model, and then generates an embedding vector for each token in the first token sequence. For example, if the user input is "abcdef", the corresponding token sequence is "a,b,c,d,e,f". The input of each layer of the neural network model is the vector representation corresponding to each token.

[0091] 302: For the attention layer of the neural network model, generate the K vector and V vector corresponding to each of the multiple tokens based on the first token sequence.

[0092] The neural network model comprises multiple attention layers, with the output sequence of the previous attention layer serving as the input sequence for the current attention layer. For any attention layer, the key matrix W obtained during training... K Value matrix W V and query matrix W QGenerate the K, V, and Q vectors of the input sequence, which means obtaining the K and V vectors generated by any attention layer, i.e., the K and V vectors corresponding to each of the multiple tokens. The neural network model includes LLM and other models that require attention calculations.

[0093] 303: Obtain the first quantization strategy corresponding to the attention layer, wherein the first quantization strategy includes a measurement method for at least one of the K vectors and V vectors generated by the attention layer.

[0094] In this embodiment, the first quantization strategy refers to the quantization strategy used by any attention layer during the inference phase of the neural network model to quantize the K-vectors and V-vectors. Specifically, because there are inter-layer differences among the attention layers of the neural network model—for example, the attention weights of the earlier attention layers differ significantly from those of the later attention layers—the K-vectors and V-vectors generated by the earlier attention layers are more irregularly distributed, while those generated by the later attention layers are more regularly distributed. Therefore, the first quantization strategy corresponding to each attention layer is different; that is, this application employs different quantization strategies for different attention layers.

[0095] For example, the first quantization strategy corresponding to any attention layer includes one or more of the following quantization parameters: the metric method of the K vector and V vector output by that attention layer; the quantization bits of the K vector; the quantization bits of the V vector; the number N (recent window size) of historical nearest neighbor tokens on which the attention value of each token depends; the number M (group size) of tokens required to perform one quantization; and the number P of key tokens to be identified to perform the quantization of the K vector and V vector output by that attention layer. It should be noted that the differences in the first quantization strategies corresponding to different attention layers can be reflected in the different categories of quantization parameters or the different values ​​of the quantization parameters in the first quantization strategies corresponding to different attention layers.

[0096] It should be noted that the metrics for the K vector and the V vector can be the same, such as both using the L2 norm, or they can be different, such as the K vector using the L1 norm and the V vector using the L2 norm. For example, the metrics for the K vector and the V vector can also be a combination of the K vector's and V vector's metrics, such as a weighted sum of the K vector's and V vector's metrics.

[0097] In one possible implementation, the first quantization strategy corresponding to any attention layer of the neural network model can be predefined by the developers. In this case, the user does not need to perform any operation and can directly use the predefined quantization parameters for quantization.

[0098] In one possible implementation, the first quantization strategy corresponding to any attention layer of the neural network model can be user-defined. The electronic device can receive third-party instructions, which specify one or more of the following quantization parameters: the metric of at least one of the K and V vectors output by any attention layer; the quantization bits of the K vector; the quantization bits of the V vector; the number N of historical nearest neighbor tokens upon which the attention value of each token depends; the number M of tokens required for one quantization operation; and the number P of key tokens to be identified for quantizing the K and V vectors output by that attention layer. The metric of at least one of the K and V vectors includes one or more of the following: L1 norm, L2 norm, L∞ norm, and Lp norm. Based on the quantization parameters indicated by the third-party instructions, the electronic device can generate the first quantization strategy for any attention layer.

[0099] In this implementation, the user can instruct the electronic device via a third user instruction to specify the metric of at least one of the key K vector and value V vector output by any attention layer. For example, the metric can be one or more of the L1 norm, L2 norm, L∞ norm, and Lp norm, so that the electronic device can subsequently use at least one of the metric methods of the K vector and V vector to identify the key token in the first token sequence.

[0100] For example, the third user instruction can be program code input by the user, an instruction input by the user through an application programming interface (API), or an instruction input by the user based on a human-computer interaction interface (HCI). The neural network model provided in this application embodiment offers an HCI during the deployment phase, allowing users to set quantization parameters for each attention layer during HCI deployment. For example... Figure 4 As shown, this human-computer interaction interface is an advanced settings interface for deploying a neural network model. This interface provides a choice of quantization methods, such as full precision and quantization. When the user selects full precision, the setting areas for each quantization parameter in the quantization strategy on the right are inoperable; when the user selects quantization, these areas become operable. The user inputs corresponding values ​​in the setting areas for quantization parameters such as the quantization bits of the K vector, the quantization bits of the V vector, the recent window size, the group size, the number of key tokens, and the measurement method, and sets the attention layer to which these quantization parameters are applied. Finally, by selecting save, the first quantization strategy for that attention layer is generated. Optionally... Figure 4In the human-computer interaction interface shown, when the user selects quantization, the default values ​​set by the developer are displayed in the setting area of ​​each quantization parameter on the right. The user can adjust all or some of the quantization parameters according to their own needs, thereby generating the first quantization strategy for the corresponding attention layer.

[0101] 304: Based on at least one vector generated by the attention layer and at least one vector measurement method, determine the first target K vector and the first target V vector that need to be quantized among multiple K vectors and multiple V vectors generated by the attention layer.

[0102] In this embodiment of the application, the electronic device can obtain the measurement value of at least one of the K vectors and V vectors output by any attention layer based on at least one of the K vectors and V vectors output by any attention layer and the measurement method of at least one vector. Based on the measurement value, it can identify the K vectors and V vectors output by any attention layer that do not need to be quantized, and identify the K vectors (i.e., the first target K vector) and V vectors (i.e., the first target V vector) output by any attention layer that need to be quantized.

[0103] For example, step 304 may include:

[0104] 3041: Determine the key token in the first token sequence obtained based on user input, based on at least one of the K vector and V vector output by any attention layer and the metric of at least one vector.

[0105] For example, in this application embodiment, the metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold. That is, tokens with abnormal metric value distributions are defined as key tokens. The information of these tokens plays an important role in inference, so their full-precision representation is usually retained without quantization. The first threshold is usually greater than the second threshold. The first and second thresholds can be set based on experience or determined based on the overall distribution of the metric values ​​of each token in the first token sequence.

[0106] In this first token sequence, the metric value corresponding to each token is determined based on at least one of the K-vector and V-vector corresponding to each token, and the metric method of at least one vector. The K-vector and V-vector output by any attention layer include the K-vector and V-vector corresponding to each token. Specifically, for any attention layer, if the metric method of the K-vector in its first quantization strategy is L1 norm and the metric method of the V-vector is L2 norm, then for any token in the first token sequence, its metric value can be determined based on the L1 norm of the token's K-vector, or based on the L2 norm of the token's V-vector, or it can be determined based on both the L1 norm of the token's K-vector and the L2 norm of the token's V-vector. For example, if the first token sequence is "a,b,c,d,e,f", then if the L1 norm of the K vectors of "d" and "f" is greater than or equal to the first threshold, or the L1 norm of the K vectors of "d" and "f" is less than or equal to the second threshold, then tokens "d" and "f" are key tokens identified by the current attention layer; or if the L2 norm of the V vectors of "d" and "f" is greater than or equal to the first threshold, or the L2 norm of the V vectors of "d" and "f" is less than or equal to the second threshold, then tokens "d" and "f" are key tokens identified by the current attention layer; or if the weighted sum of the L1 norm of the K vectors and the L2 norm of the V vectors of "d" and "f" is greater than or equal to the first threshold, or the weighted sum of the L1 norm of the K vectors and the L2 norm of the V vectors of "d" and "f" is less than or equal to the second threshold, then tokens "d" and "f" are key tokens identified by the current attention layer. It should be noted that when the K-vector and V-vector are measured using different methods, a token can have two metric values. For example, the metric value of "d" includes the L1 norm of the K-vector and the L2 norm of the V-vector. In this case, there can be two first thresholds, denoted as a1 and a2, and two second thresholds, denoted as b1 and b2. If the L1 norm of the K-vector of token "d" is greater than or equal to a1, and the L2 norm of the V-vector of token "d" is greater than or equal to a2, then token "d" is the key token identified by the current attention layer; or if the L1 norm of the K-vector of token "d" is less than or equal to b1, and the L2 norm of the V-vector of token "d" is less than or equal to b2, then token "d" is the key token identified by the current attention layer.

[0107] It should be noted that the L1 norm, L2 norm, and weighted sum are merely illustrative examples. The measurement methods in the embodiments of this application include, but are not limited to, L1 norm and L2 norm. For example, they can also be L∞ norm, Lp norm, cosine similarity with the K vector and V vector of a certain fixed token, etc.

[0108] In this implementation, tokens in the first token sequence whose metric value is greater than or equal to a first threshold, or whose metric value is less than or equal to a second threshold, are identified as key tokens. The distributions of the K-vectors and / or V-vectors of these key tokens are relatively anomalous. Quantizing their K-vectors and V-vectors may introduce significant quantization errors. Therefore, dynamically identifying these key tokens helps preserve their full-precision vector representation, thereby reducing quantization errors.

[0109] 3042: Determine the first target token in the first token sequence that needs to be quantized by K vector and V vector based on the key token.

[0110] 3043: Determine the K vector and V vector corresponding to the first target token as the first target K vector and the first target V vector.

[0111] In the embodiments of this application, such as Figure 5 As shown, after the electronic device determines the key token based on at least one of the K vector and V vector corresponding to each token and the measurement method of at least one vector, it excludes the key tokens (d and f) from the first token sequence and determines the remaining tokens as the first target tokens (a, b, c, e). The K vector and V vector corresponding to the first target token are the first target K vector and the first target V vector that need to be quantized.

[0112] For example, in a first quantization strategy that includes the number N of historical nearest neighbor tokens upon which the attention value of each token depends, the electronic device determines the first target token in the first token sequence in a sliding manner based on the size of the recent window. For the i-th token in the first token sequence whose attention value needs to be calculated at the current time step, the historical nearest neighbor token sequence upon which the attention value of the i-th token depends is determined based on N. Tokens other than key tokens among the tokens preceding the historical nearest neighbor token sequence are determined as target tokens that need to be quantized using K-vector and V-vector before the i-th token. The first target token includes the target tokens that need to be quantized using K-vector and V-vector before the i-th token. For example, in "a,b,c,d,e,f", the i-th token is "e". Assuming N=2, the historical nearest neighbor token sequence is "c,d". Tokens preceding the historical nearest neighbor token sequence "c,d" include "a,b". Key tokens are excluded from "a,b", thus obtaining the target token determined at the current time step.

[0113] It should be noted that when the i-th token is the last token in the first token sequence, after completing the attention calculation for the last token, the non-key tokens in the historical nearest neighbor token sequence on which it depends are also determined as the first target token.

[0114] In this implementation, the electronic device can perform a target token lookup operation when calculating the attention of each token. Since the first quantization strategy includes the number N of historical nearest neighbor tokens on which the attention value of each token depends, the electronic device needs to first determine the sequence of historical nearest neighbor tokens on which the i-th token depends when determining the target token. Then, it needs to determine the target token that needs to be quantized by K vector and V vector from the tokens before the historical nearest neighbor token sequence, so as to avoid the K vector and V vector of the historical nearest neighbor tokens on which the i-th token depends being quantized, thereby affecting the accuracy of the attention calculation of the i-th token.

[0115] For example, an electronic device can directly determine the first target token in the first token sequence that needs to be quantized by K-vector and V-vector based on at least one vector metric method, thereby determining the first target K-vector and the first target V-vector. For instance, the electronic device can determine the metric value corresponding to each token based on at least one of the K-vectors and V-vectors and the metric method of that vector; the metric value corresponding to each token is also the metric value corresponding to the K-vector and V-vector of each token. For any attention layer, based on this method, multiple K-vectors and multiple V-vectors can be generated, corresponding to the metric values ​​(i.e., the metric values ​​corresponding to multiple tokens). Tokens with metric values ​​less than a third threshold are determined as the first target tokens, and their corresponding K-vectors and V-vectors are the first target K-vector and the first target V-vector. Alternatively, tokens with metric values ​​greater than a fourth threshold are determined as the first target tokens, and their corresponding K-vectors and V-vectors are also the first target K-vector and the first target V-vector. For example, the third threshold can be less than or equal to the first threshold, and the fourth threshold can be greater than or equal to the second threshold.

[0116] In this implementation, the metric values ​​corresponding to multiple K vectors and multiple V vectors actually refer to the metric values ​​corresponding to multiple tokens. The electronic device can determine the metric value corresponding to each token based on at least one of the K vectors and V vectors corresponding to each token and the metric method of the at least one vector. The K vectors and V vectors corresponding to tokens whose metric values ​​are less than the third threshold or greater than the fourth threshold are determined as the first target K vector and the first target V vector, respectively. Therefore, it is not necessary to determine the first target K vector and the first target V vector based on the key token, which is beneficial to improving the recognition efficiency of the first target K vector and the first target V vector.

[0117] 305: Quantize the first target K vector and the first target V vector.

[0118] In this embodiment of the application, for the determined first target K vector and first target V vector, the electronic device can use the quantization bits of the K vector in the first quantization strategy to quantize the first target K vector, and use the quantization bits of the V vector in the first quantization strategy to quantize the first target V vector.

[0119] For example, if the first quantization strategy includes the size of the recent window and the number of tokens M (i.e., the group size) required to perform one quantization, and the number of target tokens Q that need to be quantized by K vector and V vector before the i-th token is greater than or equal to M, then the electronic device quantizes the K vector (using the quantization bits of the K vector) and V vector (using the quantization bits of the V vector) corresponding to the Q target tokens or M target tokens among the Q target tokens. Specifically, when Q is exactly equal to M, the M target tokens can be grouped together, and the K vector of the M target tokens can be quantized using the quantization bits of the K vector, and the V vector of the M target tokens can be quantized using the quantization bits of the V vector. Alternatively, when Q is greater than M, the Q target tokens can be grouped together, and the K vector of the Q target tokens can be quantized using the quantization bits of the K vector, and the V vector of the Q target tokens can be quantized using the quantization bits of the V vector. Or, when Q is greater than M, M target tokens can be selected from the Q target tokens, grouped together, and the K vector of the M target tokens can be quantized using the quantization bits of the K vector, and the V vector of the M target tokens can be quantized using the quantization bits of the V vector. Target tokens among the Q target tokens that have not undergone K-vector and V-vector quantization will undergo K-vector and V-vector quantization when the number of target tokens accumulates to be greater than or equal to M.

[0120] It should be understood that the K and V vectors output by the first few attention layers of a neural network model are relatively irregularly distributed, so a higher bit width quantization is used. However, the K and V vectors output by the attention layers after these attention layers are more evenly and regularly distributed, so a lower bit width quantization is used.

[0121] For example, for Q target tokens or M target tokens among Q target tokens, their K vector can be quantized by channel dimension or by token dimension; their V vector can be quantized by channel dimension or by token dimension. Suppose the K vectors of two tokens are K1 and K2, where K1 = [0.2, 1.5, -0.7, 2.1, 0.4, -1.3] and K2 = [1.2, 0.5, -0.6, 3.1, 0.4, -2.3]. Quantization by token dimension means that K1 determines its quantization range based on the maximum and minimum values ​​of its six elements, and then quantizes the six elements of K1 based on this range. Similarly, K2 determines its quantization range based on the maximum and minimum values ​​of its six elements, and then quantizes the six elements of K2 based on this range. Defining the number of channels as 6, each channel of K1 and K2 has one element. Therefore, each channel of K1 is [0.2], [1.5], [-0.7], [2.1], [0.4], [-1.3], and each channel of K2 is [1.2]. [0.5],[-0.6],[3.1],[0.4],[-2.3], quantization by channel dimension means that for the first channel, the quantization range of the first channel is determined based on the maximum and minimum values ​​of [0.2] of K1 and [1.2] of K2, and then quantized based on the quantization range of [0.2] of K1 and [1.2] of K2. For the second channel, the quantization range of the second channel is determined based on the maximum and minimum values ​​of [1.5] of K1 and [0.5] of K2, and then quantized based on the quantization range of [1.5] of K1 and [0.5] of K2. For the third channel, the quantization range of the third channel is determined based on the maximum and minimum values ​​of [-0.7] of K1 and [-0.6] of K2, and then quantized based on the quantization range of [-0.7] of K1 and [-0.6] of K2, and so on for subsequent channels. It should be understood that quantization by channel dimension usually considers more tokens. The two tokens mentioned above are only for illustrative purposes and do not limit the quantization by channel dimension in this application. Specifically, the quantization of V vector by token dimension can refer to the quantization of K vector by token dimension, and the quantization of V vector by channel dimension can refer to the quantization of K vector by channel dimension; these will not be elaborated further here.

[0122] In this implementation, the first quantization strategy defines the number M of tokens required to perform one quantization. For the i-th token used to calculate attention at the current time step, if the number of target tokens requiring K-vector and V-vector quantization before this token is greater than or equal to M, the electronic device is triggered to perform one K-vector and V-vector quantization. Specifically, for the K-vectors and V-vectors corresponding to Q target tokens or M of the Q target tokens, the electronic device can perform quantization along the channel dimension or along the token dimension, offering relatively flexible quantization methods.

[0123] For example, if the first quantization strategy includes the number P of any key tokens preset by the attention layer, then P is less than the length of the first token sequence.

[0124] In this implementation, for any attention layer, the number of key tokens P is limited to be less than the length of the first token sequence. This prevents the first token sequence from being identified as all key tokens, thus ensuring that non-key tokens are quantized in the current attention layer.

[0125] For example, for any attention layer, if the number of key tokens identified in the first token sequence is equal to P, and a new first key token is identified in the first token sequence, then the second key token among the P identified key tokens is replaced by the first key token; the difference between the metric value corresponding to the first key token and the decision threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the decision threshold used to determine the second key token. For example... Figure 5If "d" is identified as a key token, and the number of key tokens is exactly equal to P, and the first threshold is used to identify "d", then if "e" is also identified as a key token, and the second threshold is used to identify "e", and the difference between the metric value of "e" and the second threshold is greater than the difference between the metric value of "d" and the first threshold, it indicates that the distribution of the K vector and / or V vector of "e" is more anomalous. The information contained in this type of token is more important to the neural network model. Therefore, "d", which has a lower degree of anomalousness, can be replaced with "e", meaning that "d" is reclassified as a non-key token. Alternatively, "d" can be identified as a key token, and the number of key tokens is exactly equal to P, and the first threshold is used to identify "d". If "e" is also identified as a key token, and the threshold used to identify "e" is also the first threshold, if the difference between the measurement value of "e" and the first threshold is greater than the difference between the measurement value of "d" and the first threshold, then "d" with a lower degree of abnormality can be replaced with "e", and "d" will be re-identified as a non-key token.

[0126] In this implementation, for any attention layer, if P key tokens are identified, and a new key token is identified in the first token sequence, and the distribution of the K vector and / or V vector of this key token is more anomalous than the distribution of the K vector and / or V vector of a certain key token among the existing P key tokens, then the electronic device replaces the key token with the lower degree of anomalousness with the newly identified key token, so as to ensure that the key token with the higher degree of anomalousness is detected and the full-precision representation of such tokens is retained, thereby reducing quantization error.

[0127] For example, embodiments of this application also provide a first user interface. For any attention layer, after recognizing the key token, the electronic device will also display the first user interface, such as... Figure 6 As shown, the first user interface includes the key token finally identified by the current attention layer and its position in the first token sequence. Furthermore, the first user interface can dynamically display other key tokens identified by other attention layers and their positions in the first token sequence. For example... Figure 6 In the settings area, the user can enter the index of the attention layer and click OK. The electronic device will then display the key token identified by the attention layer corresponding to that index and its position in the first token sequence on the first user interface.

[0128] In this implementation, the electronic device can output the key tokens identified at any attention layer and their positions in the first token sequence. Because different attention layers employ different quantization strategies, especially regarding the number and measurement methods of key tokens, the key tokens identified by different attention layers will also differ. Based on the output of the first user interface, the user can intuitively perceive that the identified key tokens are dynamically changing, rather than fixed.

[0129] For example, the electronic device may also adjust the measurement method and / or P of at least one vector in the first quantization strategy based on the key token displayed in the first user interface and the position of the key token in the first token sequence.

[0130] For example, the electronic device can receive a first user instruction, which is used to adjust the measurement method of at least one vector in the first quantization strategy. For example, the electronic device can also receive a second user instruction, which is used to adjust the parameter P in the first quantization strategy. In scenarios where both the measurement method and parameter P of at least one vector in the first quantization strategy are adjusted simultaneously, the first user instruction and the second user instruction can be the same user instruction.

[0131] In this embodiment, for any attention layer, if the number of identified key tokens does not meet the user's expected number or the number set by the user, the electronic device can adjust the measurement method and / or P of at least one vector (K vector and / or V vector) in the first quantization strategy corresponding to that attention layer; or if the identified key tokens are not the key tokens that the user is interested in, the electronic device can also adjust the measurement method and / or P of at least one vector in the first quantization strategy corresponding to that attention layer. It should be noted that adjusting at least one of the measurement method and parameter P may affect the identification of key tokens. Specifically, the electronic device can adjust the measurement method and / or parameter P of at least one vector based on the key tokens displayed in the first user interface and the position of the key tokens in the first token sequence, or it can adjust the measurement method and / or parameter P of at least one vector based on user instructions (such as a first user instruction or a second user instruction). For example, increasing the value of parameter P can ensure the accuracy of model inference, but when the number of key tokens increases, the overall memory usage will also increase accordingly.

[0132] In this implementation, since the key tokens identified by each attention layer can be visualized, if the output key tokens do not meet the requirements, the electronic device can adjust the measurement method and / or parameter P of at least one vector in the first quantization strategy to further optimize the quantization strategy at the layer granularity. This is beneficial to improving the accuracy of key token identification in each attention layer, and in turn, to improving the quantization accuracy of the K vector and V vector of the target token in each attention layer.

[0133] For example, in the embodiments of this application, during the process of the electronic device processing user input through a neural network model, it can output various performance indicators, including but not limited to model throughput, model accuracy loss, compression ratio, memory usage, etc. Figure 7 As shown, the electronic device can output the memory changes when a neural network model executes processing at a certain layer, granularly. Users can enter the layer index in the layer selection area and click "OK," and the electronic device can dynamically display the memory usage of each layer. Figure 7 The horizontal axis represents time (in seconds / s), and the vertical axis represents memory usage (in gigabytes / GB).

[0134] As can be seen in this embodiment, for any attention layer, the electronic device can obtain the quantization strategy (i.e., the first quantization strategy) corresponding to that attention layer, and identify the first target K vector and the first target V vector that need to be quantized from the generated K vector and V vector based on at least one of the K vectors and V vectors output by the attention layer and the measurement method of the at least one vector in the quantization strategy, and then quantize them. For the first target K vector and the first target V vector, the quantization parameters in the quantization strategy are used to quantize them, and the introduced quantization error is usually small, which is beneficial to improving the accuracy of KV buffer quantization. As for the K vectors and V vectors other than the first target K vector and the first target V vector in the output K vector and V vector, they are generally considered to have a more abnormal distribution and contain more special information, so their full-precision representation is retained during KV quantization. Furthermore, since the quantization strategies corresponding to each attention layer are different, electronic devices can achieve a balance between compression ratio and compression effect through the conversion of inter-layer quantization strategies during the quantization process. For example, in some attention layers, the feature information may not be abundant (or the feature distribution is relatively uniform), so a high compression ratio can be used to reduce the amount of computation and memory usage. In other attention layers, the feature information may be important (or the feature distribution is relatively abnormal), so a low compression ratio is needed to improve the compression effect. The dynamic balance between compression ratio and compression effect is beneficial to improving the memory utilization of computing chips, increasing the throughput of model inference, and reducing inference latency.

[0135] For example, the key-value quantization method provided in this application can be applied to long text processing scenarios, such as automatic summarization of long texts, article generation, and complex question-answering systems. These tasks typically require models to effectively handle large-scale contexts and maintain high inference speed and accuracy without increasing memory usage.

[0136] For example, the key-value quantization method provided in this application can be applied to real-time inference service scenarios, such as online chatbots and automated customer service systems. These scenarios require rapid response to user requests and cannot use excessive memory. This application, through an adaptive quantization strategy based on key token tracking, can effectively reduce the memory requirements of the KV cache while improving response speed, making it suitable for deployment in real-time systems with limited memory and computing resources.

[0137] For example, the key-value quantization method provided in this application can be applied to large-scale model deployments. For scenarios involving ultra-large-scale language models (such as LLaMA3-405B) and batch inference, the quantization technology provided in this application can effectively improve resource utilization and reduce hardware requirements. While maintaining model performance, this application significantly reduces the storage and transmission overhead of KV caches, making it suitable for cloud service providers or large enterprises to deploy for large-scale model inference tasks.

[0138] Please see Figure 8 , Figure 8 This is a flowchart illustrating another key-value quantization method provided in an embodiment of this application. Figure 8 As shown, the method includes steps 801-810:

[0139] 801: In the process of reasoning over sample data using a neural network model, for any attention layer, obtain the second quantization strategy corresponding to that attention layer.

[0140] In this embodiment of the application, before step 801, the user needs to deploy the neural network model. This deployment operation includes uploading the model parameters. The parameters of the neural network model can come from local storage or from an online storage system, such as Object Based Storage (OBS). Figure 9A As shown, when deploying a neural network model, users can upload model parameters and set basic parameters through the basic settings interface. For example, they can enter or select the address of the model parameters, "obs: / / url", and click "Upload Model Parameters" to complete the parameter upload. They can also enter the maximum inference length supported by the neural network model and choose whether to use fp16 precision, etc.

[0141] After completing the basic settings, the model deployment enters the advanced settings interface. For example, the electronic device displays a third user interface, which is an interface for setting quantization parameters for the K and V vectors output by any attention layer. The third user interface includes one or more setting areas for at least one vector metric, N, M, and P. See [reference needed] for details. Figure 4 In the advanced settings interface, full-precision inference is used by default for any attention layer in the neural network model. Users can operate on one or more settings areas as needed, such as inputting new quantization parameter values ​​or agreeing to the default values ​​provided by the developers. The electronic device obtains a second quantization strategy based on the user's operations on one or more settings areas. For example, users can set all quantization parameters for the K vector and V vector, such as entering corresponding values ​​in the settings areas for quantization bits of the K vector, quantization bits of the V vector, recent window size, group size, number of key tokens, and measurement method, and setting the attention layer to which these quantization parameters are applied. Finally, by selecting save, a second quantization strategy for that attention layer is generated.

[0142] In this implementation, for any attention layer of the neural network model, a quantization parameter setting interface (i.e., a third user interface) for K vectors and V vectors is provided. In addition to the setting area for conventional quantization parameters such as quantization bits, this interface also includes one or more setting areas for the measurement method of at least one of the K vectors and V vectors, the setting area for N, the setting area for M, and the setting area for P. During the deployment phase of the neural network model, based on the user's operation on the one or more setting areas, the electronic device can generate an initial quantization strategy (i.e., a second quantization strategy) for any attention layer, thereby realizing the visualization of quantization parameters.

[0143] After generating the second quantization strategy, the inference system can be tested for a period of time. During the test run, users can input sample data, and the electronic device can convert it into a token sequence (i.e., the second token sequence mentioned below) through the input embedding layer of the neural network model, and then generate an embedding vector for each token in the second token sequence.

[0144] 802: Based on the second quantization strategy, the K vector and V vector corresponding to the second target token are quantized to obtain a KV cache after quantization of any attention layer; the second target token is determined based on the key tokens in the second token sequence, and the second token sequence is obtained based on the sample data.

[0145] In this embodiment of the application, the second quantization strategy also includes a measurement method for at least one of the K vectors and V vectors. The electronic device determines the key token in the second token sequence based on at least one of the K vectors and V vectors output by any attention layer and the measurement method of the at least one vector. Then, based on the key token, it determines the second target token in the second token sequence that needs to be quantized by K vectors and V vectors. Finally, it quantizes the K vectors and V vectors of the second target token based on quantization parameters such as quantization bits and group size in the second quantization strategy.

[0146] The determination of the key token, the determination of the second target token, and the quantization of the K-vector and V-vector of the second target token can be referred to respectively. Figure 3 The embodiments shown describe the determination of the key token, the determination of the first target token, and the corresponding quantization of the K vector and V vector of the first target token.

[0147] Each attention layer quantized cache includes the K vector and V vector of the key token in full precision, and the K vector and V vector of the second target token after quantization.

[0148] 803: Based on the KV cache quantized from at least one of the attention layers, determine one or more performance metrics for a neural network model to infer from sample data.

[0149] In this embodiment, one or more performance metrics include inference accuracy, compression ratio, throughput, inference accuracy loss, inference latency, and compute card utilization. Based on the description in step 802, the electronic device can obtain multiple attention layer quantized KV caches. If more attention is paid to the accuracy of model inference, the electronic device can determine the final result of model inference based on the multiple attention layer quantized KV caches, thereby obtaining the accuracy of model inference; if more attention is paid to the memory usage of the entire model inference process, the electronic device can also determine the memory usage based on the multiple attention layer quantized KV caches; if more attention is paid to the compression ratio of the current attention layer, the electronic device can determine the compression ratio of the current attention layer based on the current attention layer quantized KV cache; if more attention is paid to the compression ratio of the preceding special attention layers (including any one of the attention layers), the electronic device can determine the corresponding compression ratio based on the corresponding attention layer quantized KV caches, and so on.

[0150] 804: Provides a second user interface for displaying one or more performance metrics.

[0151] Users can determine which performance metrics have not met the requirements based on one or more performance metrics displayed on the second user interface, and then input third user commands.

[0152] 805: Receive instructions from a third user, which are used to adjust the second quantization strategy to obtain the first quantization strategy.

[0153] In this embodiment, if some or all of one or more performance metrics fail to meet the requirements, the electronic device can adjust the second quantization strategy to obtain the first quantization strategy. In other words, this embodiment allows for adjustments to the quantization strategy of the attention layer at different stages. For example, if one or more performance metrics include inference accuracy, and the inference accuracy is less than a threshold, the electronic device can adjust all or some of the quantization parameters (such as the number of key tokens, quantization bits, etc.) in the second quantization strategy to improve inference accuracy. Alternatively, the electronic device can adjust all or some of the quantization parameters in the second quantization strategy based on external input. For example, if one or more performance metrics include compression ratio, and the compression ratio is less than a threshold, the electronic device can adjust the quantization bits, the number of key tokens, etc., in the second quantization strategy, either automatically or based on external input. Based on this scheme, any attention layer can have multiple sets of first quantization strategies, each related to a performance metric that the system or user is more concerned with. For example, all or some of the quantization parameters in the second quantization strategy can be adjusted, including the quantization bits of the K vector, the quantization bits of the V vector, the recent window size, the group size, the number of key tokens, the measurement method, etc.

[0154] In this implementation, after deploying the neural network model, the user can use sample data as input to test one or more performance metrics of the neural network model's inference. During the inference process on the sample data, the electronic device identifies key tokens in the second token sequence obtained based on the sample data using a second quantization strategy corresponding to any attention layer, and quantizes the K-vector and V-vector corresponding to the second target token in the second token sequence. Based on the quantization results of at least one attention layer, the electronic device can determine one or more performance metrics of the neural network model's inference on the sample data and display one or more performance metrics through a second user interface. If one or more performance metrics do not meet the user's requirements, the electronic device can receive a third user instruction and adjust all or part of the quantization parameters in the second quantization strategy based on the third user instruction (e.g., the third user instruction includes the updated value of the quantization parameters to be adjusted) to obtain a first quantization strategy that meets different performance metric requirements. This facilitates the use of a first quantization strategy that meets different performance metric requirements for KV cache quantization in later application stages, thereby achieving a balance between the performance metrics of the neural network model's inference and compression requirements, and realizing finer-grained KV cache quantization.

[0155] For example, such as Figure 9B As shown, after the quantization parameters of each attention layer are set and the neural network model is successfully deployed, the server or system provides an Application Programming Interface (API) for users to call.

[0156] 806: Receive user input and obtain the first token sequence based on the user input. The first token sequence includes multiple tokens.

[0157] 807: For the attention layer of the neural network model, generate the K vector and V vector corresponding to each of the multiple tokens based on the first token sequence.

[0158] 808: Obtain the first quantization strategy corresponding to the attention layer, wherein the first quantization strategy includes a measurement method for at least one of the K vectors and V vectors generated by the attention layer.

[0159] 809: Based on at least one vector generated by the attention layer and the measurement method of at least one vector, determine the first target K vector and the first target V vector that need to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer.

[0160] 810: Quantize the first target K vector and the first target V vector.

[0161] The specific implementation methods for steps 806-810 can be found in the following references. Figure 3 The corresponding descriptions in steps 301-305 of the illustrated embodiments can achieve the same or similar beneficial effects.

[0162] Please see Figure 10 , Figure 10 This is a schematic diagram of a key-value quantization device provided in an embodiment of this application. This device can be applied to electronic devices or chips within electronic devices. Figure 10 As shown, the device may include an acquisition unit 1001 and a processing unit 1002; wherein:

[0163] The acquisition unit 1001 is used to receive user input and obtain a first token sequence based on the user input. The first token sequence includes multiple tokens.

[0164] The processing unit 1002 is used to generate a key K vector and a value V vector corresponding to each of the multiple tokens based on the first token sequence for the attention layer of the neural network model.

[0165] The acquisition unit 1001 is also used to acquire the first quantization strategy corresponding to the attention layer, wherein the first quantization strategy includes a measurement method for at least one of the K vector and V vector generated by the attention layer;

[0166] The processing unit 1002 is further configured to determine, based on at least one vector generated by the attention layer and at least one measurement method of the vector, a first target K vector and a first target V vector that need to be quantized among multiple K vectors and multiple V vectors generated by the attention layer; and to quantize the first target K vector and the first target V vector.

[0167] It can be seen that, Figure 10In the illustrated device, for any attention layer, the quantization strategy corresponding to that attention layer (i.e., the first quantization strategy) can be obtained. Based on at least one of the K and V vectors output by the attention layer and the metric method of that at least one vector in the quantization strategy, the first target K vector and the first target V vector that need to be quantized from the generated K and V vectors are identified and quantized. For the first target K vector and the first target V vector, the quantization parameters in the quantization strategy are used for quantization, which usually introduces a small quantization error, thus improving the accuracy of KV buffer quantization. As for the K and V vectors other than the first target K and the first target V vector in the output K and V vectors, their distribution is generally considered to be abnormal and contain special information, so their full-precision representation is retained during KV quantization. Furthermore, since the quantization strategies corresponding to each attention layer are different, a balance between compression ratio and compression effect can be achieved through the conversion of quantization strategies between layers during the quantization process. For example, in some attention layers, the feature information may not be abundant (or the feature distribution is relatively uniform), so a high compression ratio can be used to reduce the amount of computation and memory usage. In other attention layers, the feature information may be important (or the feature distribution is relatively abnormal), so a low compression ratio is needed to improve the compression effect. The dynamic balance between compression ratio and compression effect is beneficial to improving the memory utilization of the computing chip, increasing the throughput of model inference, and reducing inference latency.

[0168] In one possible implementation, regarding determining the first target K vector and the first target V vector to be quantized among multiple K vectors and multiple V vectors generated by the attention layer based on at least one vector generated by the attention layer and at least one vector metric, the processing unit 1002 is specifically used for:

[0169] Based on at least one vector generated by the attention layer and the measurement method of at least one vector, the key token among multiple tokens is determined;

[0170] Based on the key token, determine the first target token in the first token sequence that needs to be quantized by K vector and V vector;

[0171] Among multiple K vectors and multiple V vectors, the K vector and V vector corresponding to the first target token are determined as the first target K vector and the first target V vector.

[0172] In one possible implementation, the metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold; the metric value corresponding to each token in the first token sequence is determined based on at least one of the K vector and V vector corresponding to each token and the metric method of at least one vector.

[0173] In one possible implementation, the metric values ​​corresponding to the first target K vector and the first target V vector are less than a third threshold, or the metric values ​​corresponding to the first target K vector and the first target V vector are greater than a fourth threshold. The metric values ​​corresponding to the multiple K vectors and the multiple V vectors are determined based on at least one vector and at least one vector measurement method.

[0174] In one possible implementation, the first quantization strategy also includes calculating the number N of historical nearest neighbor tokens upon which the attention value of each token depends;

[0175] In determining the first target token that needs to be quantized using K-vector and V-vector in the first token sequence based on the key token, the processing unit 1002 is specifically used for:

[0176] For the i-th token in the first token sequence that requires attention value calculation at the current time step, the historical nearest neighbor token sequence on which the attention value calculation of the i-th token depends is determined based on N;

[0177] The tokens other than the key tokens in the previous token sequence of the historical nearest neighbor are identified as the target tokens that need to be quantized by K vector and V vector before the i-th token; the first target tokens include the target tokens that need to be quantized by K vector and V vector before the i-th token.

[0178] In one possible implementation, the first quantization strategy also includes the number of tokens M required to perform one quantization;

[0179] In quantizing the first target K vector and the first target V vector, the processing unit 1002 is specifically used for:

[0180] If the number of target tokens Q that need to be quantized with K vector and V vector before the i-th token is greater than or equal to M, then the K vector and V vector corresponding to the Q target tokens or M target tokens among the Q target tokens are quantized.

[0181] In one possible implementation, the first quantization strategy also includes a number P of key tokens preset by the attention layer; wherein P is less than the length of the first token sequence;

[0182] Processing unit 1002 is also used for:

[0183] If the number of key tokens determined in the first token sequence is equal to P, and a first key token is further determined in the first token sequence, then the first key token is used to replace the second key token among the P key tokens that have been determined; the difference between the metric value corresponding to the first key token and the judgment threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the judgment threshold used to determine the second key token.

[0184] In one possible implementation, the processing unit 1002 is further configured to:

[0185] Provide a primary user interface; the primary user interface is used to display the key token.

[0186] In one possible implementation, the processing unit 1002 is further configured to:

[0187] Receive a first user instruction, which is used to adjust the measurement method of at least one vector in the first quantization strategy and / or P.

[0188] In one possible implementation, the processing unit 1002 is further configured to:

[0189] Receive a second user instruction, which is used to adjust P in the first quantization strategy.

[0190] In one possible implementation, the neural network model includes multiple attention layers; the acquisition unit is also used to: during the process of reasoning over sample data using the neural network model, for any attention layer, acquire the second quantization strategy corresponding to any attention layer;

[0191] Processing unit 1002 is also used for:

[0192] The K-vector and V-vector corresponding to the second target token are quantized based on the second quantization strategy to obtain a KV cache after quantization of any attention layer; the second target token is determined based on the key tokens in the second token sequence, and the second token sequence is obtained based on sample data;

[0193] Based on the KV cache quantized from at least one of the multiple attention layers, determine one or more performance metrics for the neural network model to infer from sample data.

[0194] Provide a second user interface for displaying one or more performance metrics;

[0195] Receive instructions from a third user, which are used to adjust the second quantization strategy to obtain the first quantization strategy.

[0196] It should be noted that, Figure 10 The implementation of each unit described can also be referenced accordingly. Figures 3 to 9B The corresponding description of the illustrated embodiments. Furthermore, Figure 10 The beneficial effects of the described device can be referred to Figures 3 to 9B The corresponding descriptions of the embodiments shown will not be repeated here.

[0197] Based on the description of the above method and device embodiments, this application also provides a computing device 1100. For example... Figure 11 As shown, computing device 1100 includes: a bus 1101, a processor 1102, a memory 1103, and a communication interface 1104. The processor 1102, memory 1103, and communication interface 1104 communicate via the bus 1101. Computing device 1100 can be a server or a terminal device; for example, computing device 1100 can be... Figures 3 to 9B The electronic device in the illustrated embodiment. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0198] Bus 1101 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus 1101 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1101 may include a path for transmitting information between various components of the computing device 1100 (e.g., memory 1103, processor 1102, communication interface 1104).

[0199] Processor 1102 may include any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device.

[0200] Memory 1103 may include volatile memory, such as random access memory (RAM). Processor 1102 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DR RAM).

[0201] The memory 1103 stores executable program code, and the processor 1102 executes the executable program code to implement the functions of the aforementioned acquisition unit and processing unit, thereby achieving... Figures 3 to 9B The method of the illustrated embodiment. That is, the memory 1103 stores information for performing... Figures 3 to 9B Instructions for the method of the illustrated embodiment.

[0202] Alternatively, the memory 1103 stores executable code, and the processor 1102 executes the executable code to implement the function of the aforementioned key-value quantization device, thereby achieving... Figures 3 to 9B The method of the illustrated embodiment. That is, the memory 1103 stores information for performing... Figures 3 to 9B Instructions for the method of the illustrated embodiment.

[0203] The communication interface 1104 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0204] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. For example, the computing device can be one of the aforementioned electronic devices.

[0205] like Figure 12 As shown, the computing device cluster includes at least one computing device 1100. The memory 1103 of one or more computing devices 1100 in the computing device cluster may store the same memory for executing... Figures 3 to 9B Instructions for the method of the illustrated embodiment.

[0206] In some possible implementations, the memory 1103 of one or more computing devices 1100 in the computing device cluster may also store memory for execution. Figures 3 to 9B Partial instructions of the method of the illustrated embodiment. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for performing... Figures 3 to 9B Instructions for the method of the illustrated embodiment.

[0207] It should be noted that the memory 1103 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the key-value quantization device. That is, the instructions stored in the memory 1103 of different computing devices 1100 can implement the functions of one or more unit modules in the feature extraction unit, data segmentation unit, and control unit.

[0208] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 One possible implementation is shown. For example... Figure 13As shown, computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1103 in computing device 1100A stores instructions for executing the functions of the acquisition unit. Simultaneously, the memory 1103 in computing device 1100B stores instructions for executing the functions of the processing unit.

[0209] Figure 13 The connection method between the computing device clusters shown can be based on the fact that the key-value quantization method provided in this application requires the acquisition of quantization strategy, identification of key tokens, and KV quantization of non-key tokens. Therefore, the functions implemented by the processing unit are assigned to computing device 1100B, while the functions implemented by the acquisition unit are assigned to computing device 1100A.

[0210] It should be understood that Figure 13 The functions of computing device 1100A shown can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0211] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 12 and Figure 13 The connection method of the computing device cluster is different in that the memory 1103 of one or more computing devices 1100 in the computing device cluster can store the same instructions for executing the key-value quantization method provided in this application.

[0212] In some possible implementations, the memory 1103 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for performing key-value quantization. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for performing key-value quantization.

[0213] It should be noted that, although Figure 11 The illustrated computing device only shows bus 1101, processor 1102, memory 1103, and communication interface 1104. However, in specific implementations, those skilled in the art should understand that the computing device also includes other components necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device may also include hardware components for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device may only include the components necessary for implementing the embodiments of this application, and may not necessarily include... Figure 11 All the devices shown.

[0214] This application also provides a chip, including: a processor, configured to retrieve and run a computer program from memory, causing a device with the chip installed to perform the above-described actions. Figures 3 to 9B The method described in any of the embodiments. The chip may be a chip in a computing device.

[0215] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figures 3 to 9B The method described in any one of the embodiments.

[0216] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute... Figures 3 to 9B The method described in any one of the embodiments.

[0217] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0218] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed computing device clusters, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0222] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0223] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0224] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0225] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A key-value quantization method, characterized in that, The method includes: Receive user input and obtain a first token sequence based on the user input, the first token sequence including multiple tokens; For the attention layer of the neural network model, a key K vector and a value V vector corresponding to each of the plurality of tokens are generated based on the first token sequence; Obtain the first quantization strategy corresponding to the attention layer, wherein the first quantization strategy includes a metric method for at least one of the K vector and V vector generated by the attention layer; Based on the at least one vector generated by the attention layer and the measurement method of the at least one vector, determine the first target K vector and the first target V vector that need to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer; The first target K vector and the first target V vector are quantized.

2. The method according to claim 1, characterized in that, The determination of the first target K vector and the first target V vector that need to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer, based on the at least one vector generated by the attention layer and the metric method of the at least one vector, includes: Based on the at least one vector generated by the attention layer and the measurement method of the at least one vector, the key token among the multiple tokens is determined; Based on the key token, determine the first target token in the first token sequence that needs to be quantized by K vector and V vector; The K vector and V vector corresponding to the first target token among the plurality of K vectors and the plurality of V vectors are determined as the first target K vector and the first target V vector.

3. The method according to claim 2, characterized in that, The metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold; the metric value corresponding to each token is determined based on at least one of the K vector and V vector corresponding to each token and the metric method of the at least one vector.

4. The method according to claim 1, characterized in that, The metric values ​​corresponding to the first target K vector and the first target V vector are less than the third threshold, or the metric values ​​corresponding to the first target K vector and the first target V vector are greater than the fourth threshold. The metric values ​​corresponding to the plurality of K vectors and the plurality of V vectors are determined based on the at least one vector and the metric method of the at least one vector.

5. The method according to claim 2 or 3, characterized in that, The first quantization strategy also includes calculating the number N of historical nearest neighbor tokens upon which the attention value of each token depends; The step of determining the first target token in the first token sequence that needs to be quantized by K vector and V vector based on the key token includes: For the i-th token in the first token sequence that requires attention value calculation at the current time step, the historical nearest neighbor token sequence on which the attention value calculation of the i-th token depends is determined based on N; The tokens other than the key token among the tokens preceding the historical nearest neighbor token sequence are identified as target tokens that need to be quantized by K-vector and V-vector before the i-th token; the first target tokens include the target tokens that need to be quantized by K-vector and V-vector before the i-th token.

6. The method according to claim 5, characterized in that, The first quantization strategy also includes the number of tokens M required to perform one quantization; The quantization of the first target K vector and the first target V vector includes: If the number Q of target tokens that need to be quantized by K vector and V vector before the i-th token is greater than or equal to M, then the K vector and V vector corresponding to the Q target tokens or M target tokens among the Q target tokens are quantized.

7. The method according to any one of claims 2-3 and 5-6, characterized in that, The first quantization strategy also includes the number P of key tokens preset by the attention layer; wherein, P is less than the length of the first token sequence; The method further includes: If the number of key tokens determined in the first token sequence is equal to P, and a first key token is further determined in the first token sequence, then the first key token is used to replace the second key token among the P key tokens that have been determined; the difference between the metric value corresponding to the first key token and the judgment threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the judgment threshold used to determine the second key token.

8. The method according to any one of claims 2-3 and 5-6, characterized in that, The method further includes: A first user interface is provided; the first user interface is used to display the key token.

9. The method according to any one of claims 2-3 and 5-8, characterized in that, The method further includes: Receive a first user instruction, which is used to adjust the measurement method of at least one vector in the first quantization strategy.

10. The method according to any one of claims 2-3 and 5-7, characterized in that, The method further includes: Receive a second user instruction, which is used to adjust P in the first quantization strategy.

11. The method according to any one of claims 1-10, characterized in that, The neural network model includes multiple attention layers; before receiving user input and obtaining the first token sequence based on the user input, the method further includes: During the process of reasoning about sample data using the neural network model, for the attention layer, the second quantization strategy corresponding to the attention layer is obtained; The K-vector and V-vector corresponding to the second target token are quantized based on the second quantization strategy to obtain the KV cache after quantization of the attention layer; the second target token is determined based on the key tokens in the second token sequence, and the second token sequence is obtained based on the sample data; Based on the KV cache quantized from at least one of the multiple attention layers, determine one or more performance metrics of the neural network model for inference on the sample data; A second user interface is provided, which is used to display one or more of the performance metrics; Receive a third user instruction, which is used to adjust the second quantization strategy to obtain the first quantization strategy.

12. A device for key-value quantization, characterized in that, The device includes an acquisition unit and a processing unit; wherein: The acquisition unit is used to receive user input and obtain a first token sequence based on the user input, wherein the first token sequence includes multiple tokens; The processing unit is used to generate a key K vector and a value V vector corresponding to each of the plurality of tokens based on the first token sequence for the attention layer of the neural network model. The acquisition unit is further configured to acquire a first quantization strategy corresponding to the attention layer, wherein the first quantization strategy includes a measurement method for at least one of the K vector and V vector generated by the attention layer; The processing unit is further configured to determine, based on the at least one vector generated by the attention layer and the measurement method of the at least one vector, a first target K vector and a first target V vector that need to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer; and to quantize the first target K vector and the first target V vector.

13. The apparatus according to claim 12, characterized in that, In determining the first target K vector and the first target V vector to be quantized among the multiple K vectors and multiple V vectors generated by the attention layer based on the at least one vector generated by the attention layer and the metric method of the at least one vector, the processing unit is specifically used for: Based on the at least one vector generated by the attention layer and the measurement method of the at least one vector, the key token among the multiple tokens is determined; Based on the key token, determine the first target token in the first token sequence that needs to be quantized by K vector and V vector; The K vector and V vector corresponding to the first target token among the plurality of K vectors and the plurality of V vectors are determined as the first target K vector and the first target V vector.

14. The apparatus according to claim 13, characterized in that, The metric value corresponding to the key token is greater than or equal to a first threshold, or the metric value corresponding to the key token is less than or equal to a second threshold; the metric value corresponding to each token is determined based on at least one of the K vector and V vector corresponding to each token and the metric method of the at least one vector.

15. The apparatus according to claim 12, characterized in that, The metric values ​​corresponding to the first target K vector and the first target V vector are less than the third threshold, or the metric values ​​corresponding to the first target K vector and the first target V vector are greater than the fourth threshold. The metric values ​​corresponding to the plurality of K vectors and the plurality of V vectors are determined based on the at least one vector and the metric method of the at least one vector.

16. The apparatus according to claim 13 or 14, characterized in that, The first quantization strategy also includes calculating the number N of historical nearest neighbor tokens upon which the attention value of each token depends; In determining the first target token in the first token sequence that requires K-vector and V-vector quantization based on the key token, the processing unit is specifically used for: For the i-th token in the first token sequence that requires attention value calculation at the current time step, the historical nearest neighbor token sequence on which the attention value calculation of the i-th token depends is determined based on N; The tokens other than the key token among the tokens preceding the historical nearest neighbor token sequence are identified as target tokens that need to be quantized by K-vector and V-vector before the i-th token; the first target tokens include the target tokens that need to be quantized by K-vector and V-vector before the i-th token.

17. The apparatus according to claim 16, characterized in that, The first quantization strategy also includes the number of tokens M required to perform one quantization; In quantizing the first target K vector and the first target V vector, the processing unit is specifically used for: If the number Q of target tokens that need to be quantized by K vector and V vector before the i-th token is greater than or equal to M, then the K vector and V vector corresponding to the Q target tokens or M target tokens among the Q target tokens are quantized.

18. The apparatus according to any one of claims 13-14 and 16-17, characterized in that, The first quantization strategy also includes the number P of key tokens preset by the attention layer; wherein, P is less than the length of the first token sequence; The processing unit is also used for: If the number of key tokens determined in the first token sequence is equal to P, and a first key token is further determined in the first token sequence, then the first key token is used to replace the second key token among the P key tokens that have been determined; the difference between the metric value corresponding to the first key token and the judgment threshold used to determine the first key token is greater than the difference between the metric value corresponding to the second key token and the judgment threshold used to determine the second key token.

19. The apparatus according to any one of claims 13-14 and 16-17, characterized in that, The processing unit is also used for: A first user interface is provided; the first user interface is used to display the key token.

20. The apparatus according to any one of claims 13-14 and 16-19, characterized in that, The processing unit is also used for: Receive a first user instruction, which is used to adjust the measurement method of at least one vector in the first quantization strategy and / or P.

21. The apparatus according to any one of claims 13-14 and 16-19, characterized in that, The processing unit is also used for: Receive a second user instruction, which is used to adjust P in the first quantization strategy.

22. The apparatus according to any one of claims 12-21, characterized in that, The neural network model includes multiple attention layers; the acquisition unit is further configured to: during the process of reasoning over sample data using the neural network model, for each attention layer, acquire the second quantization strategy corresponding to the attention layer; The processing unit is also used for: Based on the second quantization strategy, the K vector and V vector corresponding to the second target token are quantized to obtain the KV cache after the attention layer quantization; The second target token is determined based on the key tokens in the second token sequence, which is obtained based on the sample data; Based on the KV cache quantized from at least one of the multiple attention layers, determine one or more performance metrics of the neural network model for inference on the sample data; A second user interface is provided, which is used to display one or more of the performance metrics; Receive a third user instruction, which is used to adjust the second quantization strategy to obtain the first quantization strategy.

23. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

24. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-11.

25. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-11.

Images