Efficient token pruning in converter-based neural networks

Abstract

The invention relates to efficient token pruning in a converter-based neural network. Key-value (KV) caching accelerates reasoning in large language models (LLMs) by allowing attention operations to extend linearly (rather than quadratically) with total sequence length. Due to the larger context length in modern LLM, the KV cache size may exceed the model size, which may negatively affect throughput. To solve this problem, a KVCrush is implemented that represents a key-value cache size reduction using head behavior similarity. KVCrush involves representing tokens using a binary vector, wherein the vector indicates which attention heads focus on tokens and which attention heads ignore tokens. Binary vectors are used in a hardware efficient low overhead process to generate representatives of unimportant tokens to be pruned without implementing k-means clustering techniques.

Description

[0001] priority This application is a non-provisional application and claims priority to and / or benefits from U.S. Provisional Application No. 63 / 716,292, filed November 5, 2024, entitled “EFFICIENT TOKENPRUNING IN TRANSFORMER-BASED NEURAL NETWORKS”, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to efficient token pruning in converter-based neural networks. Background Technology

[0003] Deep neural networks (DNNs) are widely used in various artificial intelligence fields, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of enormous computational costs. DNNs have extremely high computational requirements because they may involve a large number of operations and a large amount of data to be read and written. Summary of the Invention

[0004] According to one aspect of this application, a method for managing a key-value cache is provided, comprising: computing one or more binary vectors representing one or more tokens requesting a neural network; assigning one or more selected tokens from the one or more tokens to a bucket based on the one or more binary vectors; selecting a representative token to represent the one or more selected tokens assigned to the bucket; storing a representative key tensor and a representative value tensor computed for the representative token in the key-value cache; and providing the representative key tensor and the representative value tensor from the key-value cache to the operation of the neural network operating on the one or more selected tokens assigned to the bucket.

[0005] According to another aspect of this application, an apparatus is provided, comprising: machine-readable instructions; one or more memories storing key-value caches; and at least one computer processor that performs the above-described method when executing the machine-readable instructions. Attached Figure Description

[0006] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. To facilitate this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the drawings by way of example rather than limitation.

[0007] Figure 1An exemplary large language model is shown, implemented as a transformer-based neural network according to some embodiments of the present disclosure.

[0008] Figure 2 A serial converter block according to some embodiments of the present disclosure is shown.

[0009] Figure 3 Parallel converter blocks according to some embodiments of this disclosure are shown.

[0010] Figure 4 An attention layer of a converter block according to some embodiments of this disclosure is shown.

[0011] Figure 5 The computation in a self-attention layer without key-value (KV) caching is illustrated according to some embodiments of the present disclosure.

[0012] Figure 6 The computation in a self-attention layer with KV buffering according to some embodiments of the present disclosure is illustrated.

[0013] Figure 7 A system with distributed workers to execute requests for a converter-based neural network, according to some embodiments of the present disclosure, is shown.

[0014] Figure 8 A KV caching paging scheme according to some embodiments of this disclosure is illustrated.

[0015] Figure 9 A KV cache controller according to some embodiments of the present disclosure is shown.

[0016] Figure 10 A solution flow for efficient key-value cache management according to some embodiments of this disclosure is shown.

[0017] Figure 11 The calculation of attention weights according to some embodiments of this disclosure is illustrated.

[0018] Figure 12 A computational binary vector representation according to some embodiments of the present disclosure is shown.

[0019] Figure 13 The selection of a representative token is illustrated according to some embodiments of this disclosure.

[0020] Figure 14 This is a flowchart illustrating a method for KV cache management according to some embodiments of the present disclosure.

[0021] Figure 15 This is a flowchart illustrating a method for KV cache management according to some embodiments of the present disclosure.

[0022] Figure 16 This is a block diagram of an exemplary computing device according to some embodiments of the present disclosure. Detailed Implementation

[0023] Overview The past decade has witnessed the rapid development of data processing based on artificial intelligence (AI), especially data processing based on deep neural networks (DNNs). DNNs are widely used in fields such as computer vision, speech recognition, and image and video processing, primarily due to their ability to achieve accuracy surpassing human levels. A DNN typically consists of a series of layers. A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax, pooling, element-wise operations, linear operations, non-linear operations, etc. While DNNs are efficient in analysis and prediction, they come at the cost of enormous computational power. DNNs can consume significant amounts of electricity and runtime during training and inference.

[0024] Transducer-based neural networks, or transducer-based models, are a class of DNNs that can be used to drive large language models (LLMs) and computer vision models (referred to as ViTs in the literature). Transducer-based neural networks are used in services and applications such as natural language processing, speech processing, conversational AI assistants, image caption generation, object detection, video understanding, recommender systems, bioinformatics, time series prediction, reinforcement learning, and generative models to generate text, images, or music content. Cloud companies can provide transducer-based neural networks as managed services, where the neural networks can be served by many distributed graphics processing unit (GPU) workers, and the managed service can handle a large number of requests from a large number of users.

[0025] For some LLM or other machine learning models, autoregressive transducer-based neural networks are used. A transducer-based neural network can generate tokens one at a time (e.g., one word at a time) based on input cues and a sequence of previously generated output tokens. The process involving all operations in the transducer-based neural network is repeated token-by-total until the transducer-based neural network outputs a terminating token. A key-value (KV) cache is introduced to avoid redundant computation when generating tokens one by one. Specifically, the KV cache allows the reuse of cached key tensors and value tensors (attention outputs of operations in the transducer-based neural network) from previous tokens. The KV cache stores pre-computed key and value tensors from attention computations and allows them to be reused when generating new tokens.

[0026] Cached key and value tensors may include (intermediate) key and value tensors generated in an attention mechanism (e.g., one or more attention layers in a transformer-based neural network) during the process of generating a previously requested output token. In this document, a request refers to an instruction for a transformer-based neural network to generate one or more output tokens based on one or more input tokens. A request may include a request for a transformer-based neural network to generate one or more responses with one or more output tokens in response to an input cue having one or more input tokens. This generation may involve autoregressive generation of tokens, where generating the next token involves using an already generated token as part of an input token. A request may include or relate to one or more tokens. Cached key and value tensors may correspond to these one or more tokens. Using a KV cache to store cached key and value tensors can significantly reduce computation time and memory usage. Intermediate key and value tensors may include key and value tensors generated across layers and within attention heads during token generation.

[0027] In this paper, the input or output data of deep learning operations (e.g., the attention output or intermediate attention output of an attention mechanism in an attention layer) can be arranged into a data structure called a tensor. A tensor is a data structure with multiple elements along one or more dimensions. Examples of tensors include vectors (one-dimensional (1D) tensors), matrices (two-dimensional (2D) tensors), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and tensors with even higher dimensions. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimensions can be measured by the number of data points along an axis. The dimensions of a tensor define its shape. Attention mechanisms can produce attention outputs, such as key tensors and value tensors corresponding to one or more tokens, which can be cached in a KV cache to avoid redundant computation.

[0028] Key-value (KV) caching can accelerate inference in LLMs by allowing attention operations to scale linearly (rather than quadratically) with the total sequence length. A key challenge in executing transformer-based neural networks and serving a large number of requests to these networks is managing the KV cache. Due to the large context lengths in modern LLMs, the KV cache size can exceed the model size, negatively impacting throughput. Efficient use of the KV cache can reduce the cost of serving individual requests, increase the throughput of managed services, and improve the availability of managed services. Challenges can arise when executing neural networks against a KV cache with a limited memory budget, as is the case in most practical implementations. Managing a KV cache is not straightforward because the KV cache size grows linearly with the sequence length (which can be extremely large for each request), the sequence length is unpredictable, and the system load (e.g., the number of requests to the managed service) can be unpredictable. In some cases, the KV cache may require many times more memory than is used to store model parameters. Therefore, deploying large language models during the generation of long context tokens is challenged by high memory requirements. This is primarily due to the need to store all previous tokens in the attention module, resulting in a significant memory footprint for the KV cache.

[0029] Some solutions address this challenge by minimizing the memory footprint of the KV cache using techniques such as discarding / skipping low-attention tokens, quantization, and matrix approximation. In one set of methods, tokens are retained based on their importance. In other words, important tokens or a subset of tokens are retained in the KV cache. Importance refers to the token's significance to the attention mechanism of a transformer-based neural network, or the token's contribution to the attention mechanism. In some cases, importance can be determined based on attention weights (indicating the token's contribution) or distance from the current token. In this paper, when referring to token retention, it means that the KV cache corresponding to the token (with cached key and value tensors) is retained and stored in the KV cache. When referring to discarding / abandoning / eviction of tokens, it means that the KV cache corresponding to the token (with cached key and value tensors) is not stored or is retained in the KV cache. Some methods leverage the observation that a small subset of tokens contributes the majority of the value when calculating attention scores / weights. Some methods are based on the observation that the importance of a token decreases exponentially with increasing distance from the current token. Some token discarding schemes address this problem by eliminating tokens deemed unimportant to the attention algorithm based on factors such as attention weights or distance from the current token. However, these methods often suffer from insufficient accuracy, even with a modest reduction in the number of tokens. Another group of methods, besides preserving important tokens, focuses on preserving less important tokens by introducing noise, applying matrix approximations (e.g., approximating a large matrix with a smaller one), or reducing the precision of the data representation (e.g., implementing mixed-precision quantization). Some techniques aim to preserve all tokens through various strategies, including matrix approximation, mixed-precision quantization, etc. In some methods, a KV cache corresponding to important tokens is maintained, while a KV cache corresponding to less important tokens is strategically managed. One method introduces Gumbel noise to preserve diluted versions of unimportant tokens. Another method employs mixed-precision quantization to reduce the memory footprint of unimportant tokens. Another method utilizes low-rank matrix approximation. These groups of methods involve KV cache compression, which aims to reduce the size of the KV cache. Compression techniques can often suffer from precision loss if not properly managed.

[0030] To address one or more of these problems and achieve efficient KV cache compression, an efficient representation of key-value states and a low-overhead token pruning algorithm are introduced to reduce the size of the KV cache. The technique described in this paper can be called KVCrush, which represents a reduction in key-value cache size using head behavior similarity. K EY- V ALUE C ACHESIZE R EDUCTION U SING SIMILARITY IN H EAD-BEHAVIOR).

[0031] KVCrush provides a hardware-efficient alternative representation of tokens. The scheme leverages attention weight patterns across attention heads to generate a binary vector or binary feature vector for each token. In some embodiments, the binary vector can be computed based on an attention weight matrix calculated by the attention heads. A normalized row summation of the attention weight matrix can be computed for each attention head. This normalized row summation can be binarized using a threshold to obtain the binary vector. The binary vector has values ​​indicating which attention heads pay attention to the token and which ignore it. Furthermore, this alternative representation is much smaller than the original key tensor and value tensor or some other floating-point feature vector, but it preserves semantic information that can be used for token pruning without significantly affecting accuracy (e.g., to convey token importance and similarity to other tokens). These teachings can be extended to generate binary vectors or binary feature vectors for each KV cache page with one or more tokens.

[0032] KVCrush implements precision-aware, low-overhead token pruning (or KV cache compression) using binary vectors. KVCrush buckets tokens into different buckets (e.g., buckets can represent token groups) using alternative binary representations of the tokens and anchor vectors (e.g., representative anchors). The bucketing (or grouping) process uses low-overhead distance computation that scales linearly with the number of tokens and does not involve sorting. Different buckets can have corresponding distance ranges. If a token falls within a specific distance range corresponding to a bucket, that token is assigned to that bucket. In some embodiments, multiple static / constant anchor vectors (e.g., multiple static / constant anchors) can be used, which can improve the accuracy of grouping in some cases. After bucketing, representative tokens can be selected from the tokens assigned to a bucket to serve as proxies or representations of the tokens assigned to that bucket. Representative tokens can be selected to represent buckets. The bucketing method ensures that the overhead introduced by grouping tokens remains minimal (<1%) in the LLM inference flow. The bucketing method uses buckets in a one-off manner, efficiently producing proxies or representations of unimportant or unpruned tokens, and is significantly faster than k-means clustering techniques. The representative key and value tensors computed for representative tokens of a given bucket can be stored in a reduced KV memory footprint and used to approximate the key and value tensors computed for tokens of a given bucket. This bucketing method ensures that tokens assigned to different buckets are represented in a reduced KV memory footprint while adequately maintaining the inference accuracy of the model.

[0033] KVCrush not only surpasses state-of-the-art importance-based token retention schemes by 10% in accuracy, but it is also compatible with KV caching paging schemes and hybrid quantization schemes.

[0034] The techniques described in this paper provide a key-value (KV) cache compression scheme to improve LLM inference throughput and reduce latency without sacrificing inference accuracy. This scheme enables efficient LLM inference on central processing units (CPUs), GPUs, and hardware neural network accelerators by reducing the memory footprint of the KV cache. The scheme also maps efficiently to hardware implementations. For a given memory budget, the techniques described in this paper achieve higher accuracy than existing attention-importance-based token pruning methods. This allows enterprises to scale to longer contexts and / or increase service capacity, both of which are critical for enterprises providing solutions based on converter-based neural network models.

[0035] For a given memory budget, KVCrush delivers superior accuracy compared to state-of-the-art KV cache compression schemes alone, and also achieves better compression than other compression techniques. In one experiment, KVCrush reduced the KV cache size by 4x with less than a 1% decrease in accuracy relative to the full cache. In another experiment, KVCrush achieved state-of-the-art average accuracy with less than 0.5% overhead in total inference latency. Compared to a baseline implementation, KVCrush delivers significant accuracy improvements for state-of-the-art models when evaluated across various application scenarios, such as question answering and induction.

[0036] In some scenarios, running standard clustering solutions for KV cache compression (e.g., k-means clustering) is not feasible due to the very high clustering overhead. In an inference latency study, results show that the method described in this paper can reduce average memory access latency by decreasing memory footprint and token pruning overhead.

[0037] In some experiments using different "anchors" (e.g., random, mean, alternating one and zero), KVCrush was able to outperform the baseline accuracy for some models.

[0038] The various embodiments described herein can be illustrated within the context of a particular architecture or implementation. It is anticipated that the teachings of the embodiments described herein can be applied to other neural networks or models with attention mechanisms, where key-value (KV) caching schemes can be employed to reduce computation. It is also anticipated that the teachings of the embodiments can be applied to distributed computing systems / environments implementing KV caching. Furthermore, it is anticipated that the teachings of the embodiments can be applied to standalone computing systems implementing KV caching.

[0039] Converter-based neural networks or converter-based models Generative AI models (e.g., LLMs) are sweeping the computing industry. These models, with their massive numbers of parameters, are demonstrating state-of-the-art performance across a wide range of tasks. The current trend for LLM models is moving towards a scale of trillions of parameters. One estimate suggests that the number of models is increasing tenfold every two years. This current trajectory makes it nearly impossible for small and medium-sized players to run and service LLMs, and the sheer size of these models (a single model requiring 325GB of memory just to load its weights) renders traditional optimization techniques (e.g., prefetching, data streaming, and caching) completely ineffective. Furthermore, LLMs pose significant challenges to a platform's computational and memory resources (both bandwidth and capacity) during inference. In addition, stringent latency requirements (on the order of 50-100ms) make delivering high throughput while maintaining latency even more challenging.

[0040] Figure 1 An exemplary LLM 100, implemented as a converter-based neural network according to some embodiments of this disclosure, is illustrated. The LLM 100 may include one or more components: one or more tokenizers 104, a stack of converter blocks 110 (e.g., shown as converter block 0, converter block 1, converter block 2… converter N), one or more classifiers 112, and one or more detokenizers 114. The one or more tokenizers 104 may split input data (e.g., a prompt 102) into tokens. For example, prompt 102 may include text, and the one or more detokenizers 114 may split prompt 102 into subwords. One or more tokens (represented as X 106) may be converted into one or more embeddings 108, which include high-dimensional input features for the converter block stack 110. The converter block stack 110 may acquire knowledge about the input data.

[0041] The converter blocks in the converter block stack 110 can include two types of layers equipped with learning parameters: attention layers and feedforward (FFN) layers. Figure 2 An exemplary arrangement of the converter block is shown. Figure 3 Another exemplary arrangement of the transformer block is shown. The attention layer allows the model to weight the importance of tokens based on their contextual relevance and capture their dependencies. The attention layer implements the attention mechanism of the transformer block, which captures contextual information by focusing on positions in the sequence. The FFN layer independently provides non-linear transformations of the tokens.

[0042] One or more classifiers 112 can generate predictions or tokens based on the learned representations of the transformer block stack 110. The tokens can be used by one or more desegmenters 114 to generate generated text 116.

[0043] LLM 100 can be used as a framework to model complex relationships in text, images, audio, video, point clouds, graphics, and more. This framework can be used to expand the number of learned parameters to model even more complex relationships.

[0044] LLM 100 is designed to model sequential text in an autoregressive manner. Each subsequent token (denoted as Y 182) is determined by the context of the previous token. During the training process of LLM 100, the transformer architecture is assigned to learn to predict the next token Y 182 from text slices with known subsequent tokens. Utilizing the vast amounts of text data available on the Internet, the size of the transformer can be scaled dramatically to hundreds of billions of parameters. LLM 100 can be referred to as an autoregressive transformer, a causal transformer, a decoder-only transformer, and a decoder-only transformer. The subsequent alignment stage enables LLM 100 to engage in context-aware and human-preference-aligned dialogue. Conversational LLMs involving LLM 100 can be called Generative Pre-trained Transformers (GPTs). Aligned LLMs can be called instruction-adjusting, instruction-following, and supervised fine-tuning LLMs.

[0045] Autoregressive modeling requires sequence prediction during deployment, so LLM-based applications generally involve text generation, outputting tokens one by one. The autoregressive nature of the model means that prediction for each token involves a complete model structure. Due to the sheer number of model parameters (currently reaching billions), sequence inference is computationally demanding, characterized by initial computational intensity for the first prediction, followed by subsequent token-to-token predictions constrained by memory bandwidth. The computational complexity of the attention layer is a quadratic function of the sequence length. This complexity severely limits performance, especially for longer sequences.

[0046] Figure 2 A serial converter block 200 according to some embodiments of the present disclosure is shown. The serial converter block 200 includes an attention layer 204 and an FFN layer 206. An input X 202 is first processed by the attention layer 204, and the output of the attention layer 204 is passed to the FFN layer 206. The FFN layer 206 may produce an output X' 208. In some cases, the serial converter block 200 may include a skip connection that passes the input X 202 to be added to the output X' 208. The serial converter block 200 may be implemented as follows: Figure 1 One of the converter blocks in the stack of 110 converter blocks.

[0047] Figure 3Parallel converter block 300 according to some embodiments of the present disclosure is illustrated. Parallel converter block 300 includes an attention layer 304 and an FFN layer 306. Input X 302 is processed by attention layer 304 and then by FFN layer 306 (in parallel). The outputs of attention layer 304 and FFN layer 306 are combined at adder 308. Adder 308 may produce a sum of its inputs (e.g., the outputs of attention layer 304 and FFN layer 306). Adder 308 may produce a weighted sum of its inputs (e.g., the outputs of attention layer 304 and FFN layer 306). Adder 308 may produce a result X' 310. In some cases, parallel converter block 300 may include skip connections that pass input X 302 to be added to output X' 310. Parallel converter block 300 may be implemented as... Figure 1 One of the converter blocks in the stack of 110 converter blocks.

[0048] like Figure 1 As shown, converter-based neural network models rely on encoder-decoder stacks with the same layers. Figures 2-3 As shown, each layer can have two key components: self-attention and a feedforward network. Self-attention allows the model to analyze the entire sequence simultaneously, but a single mechanism may miss subtle differences. Multi-head attention solves this problem by creating multiple independent "heads" that focus on different aspects of word relationships. Figure 4 The multi-head attention mechanism is illustrated. The outputs from these heads are combined to gain a richer understanding. The feedforward network complements self-attention by introducing non-linearity, enabling the model to learn complex patterns. The number of layers stacked in the encoder and decoder (depth) and the number of heads within each layer (width) are hyperparameters. More layers and heads enhance the model's ability to capture long-term dependencies but increase complexity. The heads and layers of the individual transformer blocks work together to give the LLM a fine-grained understanding of textual data, thus achieving superior performance in natural language processing tasks.

[0049] Figure 4 An attention layer 400 of a converter block according to some embodiments of the present disclosure is shown. The attention layer 400 may include: Figure 1 The converter block stack 110 is a portion of the converter block. For example, attention layer 400 shows a multi-head attention layer with multiple attention heads. Input X 402 is transformed into query (Q), key (K), and value (V). Attention layer 400 includes using the query weight matrix W Q Parallel linear projection of the query 404. Attention layer 400 includes the use of key weight matrix W. K Parallel linear projection of the keys 406. Attention layer 400 includes the use of value weight matrix W. VThe values ​​are then subjected to parallel linear projection 408. The result of the linear projection is provided to a parallel attention head 410. The attention head 410 can apply an attention function using the outputs from one of the linear projections 404, one of the linear projections 406, and one of the linear projections 408. This attention function can be defined as: (Equation 1) In Equation 1, Q represents the output from one of the linear projections 404. In Equation 1, K represents the output from one of the linear projections 406. In Equation 1, V represents the output from one of the linear projections 408. Represents the scaling factor. Attention head 410 can be calculated. This generates the original attention score matrix based on the query and key. The attention head 410 can be computed. This generates an attention weight matrix, which is a normalized matrix of the original attention scores. The attention head 410 can be computed. This produces the final output, where attention weights are weighted by values ​​to form the final attention representation.

[0050] The outputs of the parallel attention head 410 can be concatenated and used with the output matrix W O The output is passed to linear projection 412. The output of linear projection 412 is the output X' 414 of attention layer 400.

[0051] The linear projection used in attention layer 400 may include multiplying the input of the linear projection with the learned weight matrix. In some cases, the matrix multiplication is followed by an optional nonlinear operation, such as an activation function.

[0052] Such as about Figure 1 The attention mechanism discussed in autoregressive transformer-based models is a significant bottleneck for performance on long sequences. A key-value cache can be provided to store previously computed key and value tensors from the attention mechanism, and these cached key and value tensors can be reused to generate the current token, thus avoiding intensive repetitive computation of the key and value tensors of previous tokens. The key-value cache becomes a practical optimization of the inference process to accelerate the generation throughput of LLMs, allowing attention operations to scale linearly (rather than quadratically) with the total sequence length. Figure 5 and Figure 6 The computation in attention layers with and without KV cache was compared.

[0053] Understanding Key-Value Caching and Key-Value Caching Paging Schemes Figure 5The computation in a KV-less self-attention layer according to some embodiments of this disclosure is illustrated. The self-attention layer may be part of a multi-head self-attention layer. In some embodiments, the self-attention layer is in the decoder of the converter. In some embodiments, the self-attention layer may be in the converter block, for example, Figure 1 The converter block is shown. Computation in the self-attention layer may include a multiplication of query matrix 510 with key matrix 520 (having one or more key tensors), which generates an attention weight matrix 530. Computation in the self-attention layer also includes a multiplication of attention weight matrix 530 with value matrix 540 (having one or more value tensors), which generates an output matrix 550 encoding a new token (e.g., token 5). In some cases, output matrix 550 may include a context-aware attention representation weighted by value matrix 540. Output matrix 550 may be generated by the attention layer using query matrix 510, key matrix 520, and value matrix 540 according to Equation 1, from which one or more new tokens may be generated. In other embodiments, computation in the self-attention layer may include other computations, such as computations utilizing scaling functions, SoftMax functions, etc. For simplicity and illustration, Figure 5 These calculations are not shown in the figure.

[0054] Each of the query matrix 510, key matrix 520, and value matrix 540 may include a tensor (e.g., a vector) for each token in the input sequence. For illustration and simplicity, the input sequence has four tokens: tokens 1-4. Query matrix 510 may include four query tensors generated based on the four input tokens: query tensors 1-4. Key matrix 520 may include four key tensors: key tensors 1-4. Value matrix 540 may include four value tensors: value tensors 1-4. Figure 5 In this embodiment, since the decoder does not implement KV caching, calculations are required to generate the key tensors in the key matrix 520 and all value tensors in the value matrix 540. Some calculations have already been performed in previous inference stages, such as calculations to generate key tensors 1-3 and calculations for value tensors 1-3. Repeating these calculations may be a waste of computational resources (e.g., power, time, etc.).

[0055] Figure 6 The computation in a self-attention layer with a KV cache according to some embodiments of this disclosure is illustrated. Figure 5 The implementation methods are different. Figure 6The decoder implements a KV cache. Using the KV cache, key and value tensors computed in previous inference phases (e.g., key and value tensors corresponding to tokens 1-3) are cached in the KV cache and can be reused in the current inference phase. The KV cache stores previously computed key and value tensors computed for one or more tokens in the attention mechanism and reuses them to generate the next attention output or token. In an implementation where a distributed GPU worker executes the neural network operation, the KV cache can be allocated in GPU memory, and the contents of the KV cache can be loaded from CPU memory. In an implementation where a processor executes the neural network operation, the KV cache can be allocated in one or more processor-local memories. When using a KV cache, execution time scales more smoothly with increasing sequence length. For example, intermediate KV tensors corresponding to previous tokens can be stored in the KV cache.

[0056] exist Figure 6 In the current inference phase shown, cached key tensors and value tensors can be retrieved from the KV cache. Figure 6 The dotted pattern highlights the data that can be retrieved from the KV cache. Key tensors 1-3 are retrieved from the KV cache. Value tensors 1-3 are retrieved from the KV cache. In the current inference phase, query matrix 510 can be multiplied by the concatenation of key tensor 4 and cached key tensors 1-3, followed by a SoftMax of the entire original attention score. The attention weights generated by performing the SoftMax of the original attention score can be further multiplied by the concatenation of value tensor 4 and cached value tensors 1-3 to generate new results. After inference is complete, key tensor 4 can be added to the KV cache. In some cases, key tensors 1-3 can be updated in the KV cache. Additionally, value tensor 4 is added to the KV cache. In some cases, value tensors 1-3 can be updated in the KV cache. This process is repeated for each token. KV caching can reduce the computational cost in the self-attention layer. The computational cost is significantly reduced when cached key and value tensors can be reused to generate the next token. Therefore, computational resources can be saved. The performance and efficiency of the transformer model can be improved through KV caching.

[0057] When using a KV cache, previously computed key-value tensors are stored in memory (e.g., a KV cache) to avoid redundant key-value projection computations in the attention mechanism, such as... Figure 6 As shown. The total memory usage of the KV cache instance can be easily calculated using Equation 2: (Equation 2) Accuracy ( ) is the number of bytes per stored value (e.g., B for FP32). Indicates the number of layers in the model. Indicates the dimension of the embedding. The length of the context in the token. The batch size, and the factor of 2, are imposed because two matrices are needed for the key (K) and value (V).

[0058] As shown in Equation 2, the KV cache size scales linearly with the (maximum) sequence length of the input context and the batch size. In practice, the KV cache size can be extremely large. For example, a converter-based model with 175 billion parameters might consume approximately 325 GB of memory to store the parameters. Meanwhile, with a batch size of 128 and a sequence length of 8K, the KV cache could have a memory size of approximately 4608 GB, which is several orders of magnitude (12 times) larger than the model weights themselves. Since the total sequence length cannot be known in advance, the KV cache memory requirements are therefore unknown, making LLM memory management particularly challenging. Typically, the maximum sequence length (usually 4K and growing rapidly) is used for memory allocation to host the KV cache, resulting in severely fragmented memory and very low batch sizes, making it feasible for a low number of concurrent users for LLM services.

[0059] The size of the KV cache is becoming increasingly prominent and is one of the key factors contributing to the very high cost of LLM model deployment. Reducing the KV cache memory footprint without compromising accuracy is challenging. With extending sequence length becoming a critical requirement for many enterprises, limiting the context sequence is not feasible. According to Equation 2, the only design knob available for scaling considerably large LLM deployments is the batch size (…). Reducing the batch size essentially decreases the model's throughput, thus severely degrading the total number of requests per second that the model can serve.

[0060] A key-value (KV) cache paging scheme is introduced to improve memory management and increase service model throughput. The KV cache paging scheme manages attention key and value tensors by dividing them into smaller, more manageable blocks (referred to as KV cache pages in this paper). The GPU worker's physical KV cache is divided into fixed-size blocks. Computed key and value tensors can be organized into KV cache pages. A KV cache page can have (computed) key and value tensors for a fixed number of tokens. KV cache pages can be copied within fixed-size blocks of the GPU worker's physical KV cache. The KV cache paging scheme involves storing KV cache pages within fixed-size blocks of the GPU worker's physical KV cache. KV cache pages with key and value tensors corresponding to a token sequence can be stored in non-contiguous fixed-size blocks of the GPU worker's physical KV cache. The GPU worker can be instructed, via instructions sent by a scheduler (e.g., implemented on the CPU), to retrieve cached key and value tensors of KV cache pages at a specific block in the physical KV cache when performing attention computation to generate the next token. GPU workers can be instructed to retrieve data from the KV cache based on memory addresses provided by the KV cache manager. The KV paging scheme is analogous to virtual memory and paging in an operating system. When servicing an LLM, a centralized scheduler (e.g., in a CPU-based computing system) manages the KV caches of distributed workers (e.g., GPU workers) and coordinates the execution of requests by the distributed workers. Specifically, the centralized scheduler controls where KV cache pages are stored on the physical KV caches of the distributed workers. When a distributed worker is instructed to execute a request, the centralized scheduler coordinates the swapping in and out of KV cache pages and makes the cached key and value tensors available to that distributed worker.

[0061] Figure 7 A system with distributed workers to perform requests for transducer-based neural networks is illustrated according to some embodiments of the present disclosure. The distributed workers can be requested by a centralized scheduler to perform one or more operations of the transducer-based neural network. The centralized scheduler can schedule one or more requests to be completed by the workers. The system may include a CPU 702 and one or more distributed workers (an example is shown as a GPU worker 730) communicatively coupled to the CPU 702. The CPU 702 (e.g., a computing processor, computing system, CPU-based computing system, or...) Figure 12 The computing device 1600 may be included to implement the functionality of a centralized scheduler. The GPU worker 730 (e.g., a computing processor, computing system, GPU-based computing system, hardware neural network accelerator-based computing system, or...) Figure 12The computing device 1600 shown may include operations on a converter-based neural network in response to a request from a centralized scheduler. The GPU worker 730 may serve, complete, or fulfill one or more requests.

[0062] CPU 702 may include a scheduler 704, a KV cache manager 706, and memory 712 coupled to or located locally on CPU 702. GPU worker 730 may include compute 732 and KV cache 734 coupled to or located locally on compute 732.

[0063] Scheduler 704 can coordinate the execution of requests by distributed GPU workers (e.g., GPU worker 730). Requests may include requests to perform one or more operations on a converter-based neural network. The coordination performed by scheduler 704 may be based on resource availability on GPU worker 730 and the converter-based neural network operations to be performed by the distributed GPU workers.

[0064] KV cache manager 706 can manage the KV cache (e.g., KV cache 734) of a distributed GPU worker in a paging manner. KV cache manager 706 can manage KV cache 734 via instructions sent by scheduler 704. Instructions can be included in requests sent to and executed by computation 732 of GPU worker 730 (scheduled by scheduler 704). KV cache manager 706 enables the storage of key tensors and value tensors for token sequences in non-contiguous memory space of GPU worker 730. The cache engine (not shown) of GPU worker 730 can allocate contiguous blocks coupled to computation 732 or local memory of computation 732 for use as KV cache 734. The cache engine can then divide KV cache 734 into fixed-size blocks. Blocks in the physical KV cache 734 can store key tensors and value tensors for a fixed number of tokens (referred to as the KV block size). Similarly, the memory engine (not shown) of CPU 702 can allocate memory block 712 in CPU 702 for use by GPU worker 730, or allocate memory block 712 in memory 712 for a specific request.

[0065] KV cache manager 706 may include a page-to-block table 708. Similar to virtual memory in an operating system, KV cache manager 706 can map requested logical KV cache pages to blocks in physical KV cache 734 in the page-to-block table 708. The page-to-block table 708 may include a table storing the mappings between logical KV cache pages for each request and blocks in KV cache 734. In some embodiments, entries in the table may record the location, memory address, or identifier of the physical block in KV cache 734 corresponding to the logical KV cache page. In some embodiments, entries may also include the number of padding locations or space availability within the block. Thus, consecutive logical KV cache pages corresponding to a token sequence can be mapped to non-consecutive blocks in KV cache 734, but the computation 732 performing a specific request can access KV cache 734 as if the memory were contiguous. Furthermore, separating logical KV cache pages from physical blocks allows KV cache manager 706 to expand KV cache 734 without pre-reserving large blocks of memory on GPU worker 730 (for the maximum possible sequence length).

[0066] Scheduler 704 enables one or more KV cache pages to be streamed to GPU worker 730 for swapping in or storing in KV cache 734. Scheduler 704 may include information (e.g., a page-to-block table for a specific request in page-to-block table 708) in instructions or messages destined for GPU worker 730 to allow computation 732 to retrieve logical KV cache pages from KV cache 734 at the appropriate memory location / address / block identifier when the request is satisfied. During attention computation performed by computation 732, computation 732 may be instructed by instructions sent by scheduler 704 to retrieve cached key and value tensors at one or more specified locations / addresses in KV cache 734. Figure 8 A more detailed diagram of the KV caching paging scheme is provided.

[0067] As the number of request and output tokens increases, the KV cache 734 may run out of physical KV blocks and be unable to store newly generated key tensors and value tensors. Block allocation 710 can be implemented in the KV cache manager 706 to perform eviction and swapping. Block allocation 710 can implement an eviction policy to determine whether to evict a specific KV cache page from the physical KV blocks in the KV cache 734. Block allocation 710 can copy one or more evicted KV cache pages (e.g., requested KV cache pages) to memory 712, for example, to swap space allocated in memory 712 for a specific request. Block allocation 710 can manage or track KV cache pages swapped to memory 712. If the KV cache 734 runs out of space, block allocation 710 can select a set of KV cache pages associated with the requested token sequence for eviction and move these KV cache pages from the KV cache 734 to memory 712. When computation 732 completes other incomplete requests and releases the KV cache pages of completed requests from KV cache 734, the KV cache pages that were previously evicted and copied to memory 712 can be swapped in (copied back to KV cache 734) to allow requests to be completed using KV cache pages in KV cache 734.

[0068] Figure 8 This illustrates a KV cache paging scheme according to some embodiments of the present disclosure. GPU workers (e.g., Figure 7 The GPU worker (730) can receive a request to generate one or more output tokens in response to the input prompt: "Some people favor calico cats for their". One or more output tokens may include: "fur". “patterns” … like Figure 8 As shown, the KV cache 734 of a GPU worker executing one or more requests can store contiguous key tensors and value tensors in non-contiguous memory space. A particular request may have one or more logical KV cache pages, and a page-to-block table associated with that particular request. The KV cache 734 can be divided into fixed-size blocks, such as block 0, block 1, block 2, block 3, block 4, block 5, block 6, block 7, and block 8. Fixed-size blocks can store key tensors and value tensors for a fixed number of tokens. Figure 8 As shown in the example, a block in KV cache 734 can store key tensors and value tensors for four tokens. The cached key and value tensors for this specific request can be organized into contiguous, fixed-size logical KV cache pages, such as page 0, page 1, and page 2. Logical KV cache pages can correspond to a fixed number of tokens (e.g., ...). Figure 8 The example uses key tensors and value tensors for the four tokens. As shown, pages 0, 1, and 2 are not contiguous on the physical KV cache 734.

[0069] The different parts of the KV cache paging process are shown by circles numbered 1, 2, and 3.

[0070] In Part 1 (pre-filling operation), the input prompt has seven tokens, KV cache manager ( Figure 7 The KV cache manager 706 can map two logical KV cache pages (page 0 and page 1) to physical KV blocks 3 and 2 of the KV cache 734, respectively. This mapping can be stored in a page-to-block table (e.g., ...). Figure 7 The page-to-block table (708) stores the key and value tensors computed for the seven tokens (e.g., "Some people favor calico cats for their"). These tensors can be generated by the GPU worker using an attention mechanism. The key and value tensors computed for the first four tokens (represented as page 0) are stored in block 3, and those computed for the last three tokens (represented as page 1) are stored in block 2. Note that block 2 has unused slots or free locations. The page-to-block table stores the mapping from page 0 to block 3, and from page 1 to block 2. The page-to-block table can store the number of locations filled for the corresponding logical KV cache pages (e.g., 4 locations for page 0 and 3 locations for page 1).

[0071] In Part 2 (Autoregressive Decoding Operation), the output token "fur" is generated using the cached key and value tensors stored in Blocks 3 and 2. Since there is still one slot / position available in Page 1, the key and value tensors generated for the output token "fur" are stored in Block 2. The number of positions filled for Page 1 in the page-to-block table can be updated to reflect that all four positions are now filled for Page 1.

[0072] In section 3 (another autoregressive decoding operation), another output token "patterns" is generated using the key and value tensors stored in the caches of blocks 3 and 2. Since no slots / positions are available in page 1 (or page 1 is full), a new logical KV cache page (page 2) is designated to store the key and value tensors generated for the output token "patterns". A new physical KV block (block 6) can be allocated in KV cache 734 to store page 2. The page-to-block table stores the mappings from page 0 to block 3, page 1 to block 2, and page 2 to block 6. The number of positions in the page-to-block table filled for page 2 can be set to reflect the number of positions filled for page 2.

[0073] During attention computation in autoregressive decoding operations, the required cached key and value tensors can be efficiently retrieved (from global shared memory, such as...). Figure 7The cached key and value tensors (712) are fetched into the GPU worker's local memory (e.g., KV cache 734). The GPU worker can then use the cached tensors in the GPU worker's local memory to perform attention mechanism operations and satisfy requests. The cached key and value tensors can remain in the GPU worker's local memory until the request is fulfilled by the GPU worker or until the cached key and value tensors are evicted / swapped.

[0074] As more output tokens and their key and value tensors are generated, and physical KV blocks in KV cache 734 are dynamically allocated to store logical KV cache pages (new physical KV blocks are allocated only when previous physical KV blocks are full), the KV cache paging scheme limits memory waste and can efficiently utilize all memory. Since each request will not require a maximum context length, this method allows different requests across batches to share memory (thus improving high batch sizes and throughput), which reduces memory waste and memory fragmentation in KV cache 734.

[0075] Related work Techniques for reducing the size of the KV cache in converter-based LLMs can be categorized into three types based on their handling of the key-value cache: (1) quantization, (2) sharing, and (3) eviction.

[0076] Quantization is the process of reducing the precision of model parameters and activations to decrease bit width and save memory and computational resources. Quantization can be categorized into post-trained quantization (PTQ) and quantization-aware training (QAT), with PTQ often preferred for large language models due to its lower resource requirements. Quantizing queries, keys, and values ​​to INT8 enables efficient attention operations, but these methods ignore token importance, potentially degrading generation quality, and lack detailed analysis of the impact of KV cache compression on output quality. Some methods employ mixed-precision quantization to assign different bit widths to various model components or tensors, achieving more efficient compression. These methods leverage insights into the varying sensitivities of different parts of the model to different levels of quantization.

[0077] Multi-Query Attention (MQA) and Grouped Query Attention (GQA) are techniques developed to address memory footprint issues in transformer models by sharing key-value caches across attention heads. MQA reduces memory usage by sharing key and value representations across all attention heads, improving memory efficiency and inference speed, but potentially degrading performance and generation quality. GQA extends this concept by grouping query heads to share key-value pairs, balancing memory reduction with better performance preservation. However, GQA involves higher training costs and complexity. Both approaches offer a trade-off between memory efficiency and model performance, with MQA prioritizing memory savings and GQA providing a more balanced approach.

[0078] With the rise of LLM and long context models, methods for pruning or evicting key-value pairs from the cache after input processing have attracted considerable attention, aiming to improve decoding efficiency. Evicting key-value pairs reduces memory consumption, facilitates support for larger batch sizes and context windows, and also accelerates decoding by limiting the number of key-value pairs retrieved during attention computation. Different strategies for key-value pruning focus on aggregating the attention received by each key-value pair, with low-attention key-value pairs being evicted. Some methods employ attention-based metrics to determine eviction, e.g., by summing attention weights or considering attention frequencies exceeding a threshold. One technique combines these methods with heuristic strategies for specific tokens. However, these methods limit their overall efficiency, especially for long contexts, due to the overhead introduced during pre-filling caused by the global memory requirements of attention aggregation. To address this issue, one technique limits the observation window to the final token of the input prompt, reducing complexity from O(L... 2 The computation time is reduced to O(L) while max pooling is used to preserve adjacent key-value pairs. One technique extends this particular technique by configuring a variable eviction rate across layers, where more aggressive pruning is implemented in later layers where attention is less uniformly distributed. These methods aim to balance pre-padding with decoding time performance, optimizing memory usage and computation in long-context models.

[0079] KVCrush: Efficient Tokenization Using Alternative Key-Value Representations and Weak Clustering Algorithms in Transformer-Based Neural Networks Trim to select the agent for KV cache compression Key-value (KV) caching compression methods typically rely on attention weights as an indicator of importance. However, discarding tokens with low attention weights alters the token distribution of the input sequence, leading to a decrease in accuracy. KVCrush addresses this issue by selecting representative tokens and prioritizing high-importance tokens.

[0080] Figure 9A KV cache controller 912 according to some embodiments of this disclosure is illustrated. Computational core 902 may include one or more processors to perform one or more operations of a neural network (e.g., a transformer-based neural network) or LLM. Requests with one or more tokens can be submitted to the neural network for processing. One or more operations may include the operation of one or more attention heads of the neural network. Figure 4 The operations are described within the context of [the description of the operation]. Computational core 902 may be communicatively coupled to one or more memories to store data that can be used to perform one or more operations. One or more memories may include a KV cache 904, which may store cached key and value tensors (e.g., attention outputs of attention head operations in a transformer-based neural network) and allow them to be reused by computational core 902. Figures 6-8 The context describes the KV cache.

[0081] KV cache controller 912 may be communicatively coupled to compute core 902 and / or KV cache 904 to manage KV cache 904. KV cache controller 912 may execute instructions to perform operations related to KV cache management. KV cache controller 912 may include one or more of the following: cache budget partitioning 914, token importance checker 916, token pruner 918, and representative token selector 920.

[0082] Cache budget partitioning 914 determines the overall budget for the KV cache 904 (e.g., the amount of memory allocated for storing key tensors and value tensors in the cache). Cache budget partitioning 914 determines the budget for storing key / value tensors for important tokens, and the budget for storing representative key / value tensors for unimportant tokens.

[0083] The token importance checker 916 assesses the importance of tokens and determines whether a token is important or unimportant. Unimportant tokens are pruned or evicted to compress the KV cache. The token importance checker 916 determines / assesses whether a token is important. A token can be part of a request to a neural network (e.g., a transformer-based neural network). The token importance checker 916 can determine that a token is important based on its importance score or importance. The token importance checker 916 can determine that a token (otherwise) is unimportant based on its importance score or importance. The token importance checker 916 can determine the token's importance score or importance based on attention weights or other metrics associated with the token. Figure 4As shown, the attention mechanism of the neural network's attention head, executed by computation core 902, operates to determine the attention weight matrix based on the query tensor and key tensor corresponding to different tokens. The attention weight matrix can be arranged with rows corresponding to different query tokens and columns corresponding to different key tokens (or columns corresponding to different query tokens and rows corresponding to different key tokens). Referring to Equation 1, the attention weight matrix can include computation... , The token importance checker 916 can utilize the computed attention weight matrix to determine the attention weight corresponding to a specific token. The attention weight provides an indication of the token's contribution / importance to the attention mechanism, or how much attention the attention mechanism gives to a specific token. In some cases, the contribution / importance (the attention weight corresponding to a specific token) can be inversely biased with the distance from the specific token to the currently being processed token. The token importance checker 916 can determine whether a token is important or unimportant based on its attention weight. In some embodiments, determining whether a token is important or unimportant includes comparing the attention weight corresponding to the token with a threshold. The threshold can be a hyperparameter set based on a probability distribution of the attention weights. The threshold can be set to classify a specific percentage of tokens as important and a specific percentage of tokens as unimportant. A token can be determined as important in response to its attention weight crossing or exceeding the threshold. A token can be determined as unimportant in response to its attention weight being less than the threshold.

[0084] In response to the token importance checker 916 determining that a token is important, one or more key tensors and one or more value tensors computed by one or more attention heads corresponding to the important token may be retained by the KV cache controller 912. The one or more key tensors and one or more value tensors may be stored in the KV cache 904. The cached key tensors and cached value tensors may be provided to the computation core 902 when it generates one or more other tokens to facilitate reuse.

[0085] In response to the token importance checker 916 determining that a token is unimportant or will be pruned, one or more key tensors and one or more value tensors corresponding to the unimportant token are pruned by the token pruner 918. The KV cache controller 912 may utilize one or more representative key tensors and one or more representative value tensors selected by the representative token selector 920 as proxies or representatives, instead of the originally computed key and value tensors. The key and value tensors corresponding to the unimportant token may be evicted or discarded by the token pruner 918.

[0086] The representative token selector 920 can implement an algorithm using a bucketing strategy to select representative tokens to represent token buckets or groups of tokens. The representative token selector 920 can compute one or more binary vectors representing requests to the neural network for one or more tokens based on one or more attention weight matrices computed by one or more attention heads in the neural network. The representative token selector 920 can compute binary vectors for each token or each token page. The representative token selector 920 can assign one or more tokens to one or more buckets based on one or more distances from one or more binary vectors to anchor vectors. The representative token selector 920 can select representative tokens from the one or more tokens assigned to buckets in the one or more buckets.

[0087] Token pruner 918 may store representative key tensors and representative value tensors computed by one or more attention heads of a neural network for representative tokens in KV cache 904 based on the results of representative token selector 920. When token importance checker 916 determines that a token among one or more tokens will be pruned based on the token's importance, and token pruner 918 determines that a token is assigned to a bucket, token pruner 918 may (from KV cache 904) provide the representative key tensors and representative value tensors to computational core 902 (e.g., computational logic) performing one or more operations of one or more attention heads. The representative key tensors and value tensors approximate the key tensors and value tensors computed by one or more attention heads for tokens. Token pruner 918 may prune or evict non-representative tokens, or key tensors and value tensors computed for non-representative tokens.

[0088] Figure 10 This illustrates a solution flow for efficient key-value cache management according to some embodiments of this disclosure. A key-value cache manager (e.g., Figure 9 The KV buffer controller 912 can achieve Figure 10 The solution process shown includes one or more parts for managing KV caches (e.g., Figure 9 KV cache 904).

[0089] The cache budget C can include multiple tokens, and the KV cache corresponding to these tokens can be stored in the KV cache. For example, the cache budget C can include 2048 tokens, meaning that the KV cache corresponding to 2048 tokens can be stored in the KV cache, or the KV cache has a capacity to hold 2048 tokens. In 1002, cache budget partitioning can split the cache budget C into two parts: C = C IMPORTANT + C REPRESENTATIVE (Equation 3) C IMPORTANT This can represent the number of important tokens that will be retained in the KV cache. C REPRESENTATIVEThis can represent the number of representatives / proxy members of the cached key tensor and value tensor that will be stored in the KV cache to represent unimportant tokens.

[0090] In 1004, given inputs Q, K, and V, it is possible, for example, by calculating To calculate attention weights, these attention weights can be arranged into an attention weight matrix.

[0091] In 1006, one or more technologies can be used (e.g., implementing regarding...). Figure 9 The token importance checker (as described in technique 916) selects important tokens (e.g., C). IMPORTANT (Number of tokens).

[0092] In 1008, for unimportant tokens, the KVCrush technique can be used to determine C. REPRESENTATIVE The quantity of tokens or proxies, KVCrush technology in Figures 11-13 Further explanation is needed. (C) REPRESENTATIVE The tokens representing the quantity are stored in the KV cache.

[0093] In version 1010, token pruning is implemented to compress the KV cache. Non-important and non-representative tokens are not retained in the KV cache and can be evicted. Important tokens are retained in the KV cache. Representative tokens are stored in the KV cache.

[0094] Reference Figure 9 A cache budget allocation of 914 can achieve this. Figure 10 As shown in 1002. The token importance checker 916 can implement... Figure 10 As shown in Figure 1006, the token trimmer 918 can achieve [the following]. Figure 10 The number 1010 shown represents the token selector 920, which can implement... Figure 10 1008 in the middle.

[0095] One goal of KVCrush is to replace large floating-point feature vectors (whose size is the embedding length) with compact binary feature vectors (whose size is equal to the number of attention heads, or the number of attention heads multiplied by the page size). Binary vectors, or binary feature vectors, advantageously preserve sufficient semantic information to distinguish tokens for efficient grouping and expulsion. Based on insights into the unique structural patterns exhibited by different attention heads associated with different attention heads in a neural network, KVCrush constructs binary vectors that concisely and efficiently represent or identify the attention weight patterns of tokens or token pages. Therefore, each token is represented by a concise binary vector derived from the attention weights of the corresponding head.

[0096] Binary vectors, or binary feature vectors, are a hardware-efficient alternative representation of tokens. Instead of using large floating-point feature vectors (the size of which is the embedding length), KVCrush uses small binary feature vectors (the size of which is the number of attention heads, or the number of attention heads × the page size if KV cache paging is used), which retain enough semantic information to distinguish tokens for efficient token pruning. An insight is that different attention heads located in different layers, or different attention heads in the same layer of a multi-head attention layer, often exhibit different structural patterns. Different attention heads typically have different structures. Within a given layer, each attention head focuses on a specific type of token and ignores other types. Leveraging this insight to identify attention weight patterns for different tokens or token pages, KVCrush uses (short) binary vectors based on the attention weights of each head to determine a hardware-efficient representation for each token or token page. This can be achieved by... Figure 9 The token selector 920 applies several operations to generate a binary vector. These operations are performed in... Figures 11-12 shown in . Specifically, Figures 11-12 The diagram illustrates a scenario where S tokens (denoted as T0, T1, ... TS) can be processed by H attention heads in a converter-based neural network.

[0097] Figure 11 The computation of attention weights according to some embodiments of this disclosure is illustrated. Each attention head computes an attention weight matrix using a query matrix and a key matrix. The attention operator of the attention head (implementing the computation as shown herein as Equation 1) can be applied to the shown “keys” and “queries” to produce the shown “attention weights”. Note that the attention weight matrix computation has been performed by the neural network during inference. Therefore, the attention weight matrix is ​​available without recomputation.

[0098] Figure 12 A computational binary vector representation according to some embodiments of this disclosure is shown. The attention weight matrix is ​​calculated or estimated from H attention heads, such as... Figure 11As shown. Normalized row sums of attention weight matrices in one or more attention weight matrices can be computed. Normalized row sums can be computed for each row of each attention weight matrix, and for each token of each attention weight matrix. A row-wise normalized sum can represent a token-wise normalized value for a given attention weight matrix computed by the attention heads. Normalized row sums are computed for each attention weight matrix produced by H attention heads. The results include H sets of row-wise normalized sums for S tokens, or H vectors of length S with normalized row sums, as shown in 1202. Here, the normalized row sums transform the attention weight matrix by scaling the sum of attention weights in each row so that the total row sum is exactly 1. This process involves first computed the row sum of all attention weights in a given row, and then dividing each row sum by the sum of all row sums (or the sum of all attention weights). For a given attention weight matrix, the result of calculating the normalized row sum is an S×1 column vector, where each element of the vector represents the normalized proportion of the sum of each row relative to the sum of the attention weight matrix.

[0099] When calculating the normalized row summation for each of the H attention weight matrices, the H number of S×1 column vectors can be rearranged into a matrix of size S×H, as shown in 1204.

[0100] The normalized row sums can be binarized using a threshold to produce S binary vectors, or an S×H matrix, as shown in Figure 1206. These binary vectors, also referred to herein as binary feature vectors, represent tokens. The normalized row sums of the matrix illustrated in Figure 1204 can be binarized using a threshold. The S binary vectors can include binary values ​​for each of one or more attention heads, or H binary values ​​corresponding to H attention heads. The binary values ​​are selection bits, indicating whether a given attention head focuses on or ignores a specific token.

[0101] In some embodiments, a per-attention-head threshold (or a column of a matrix illustrated as 1204) can be applied to the row summation of the column to produce a binary value, or a selection bit. Applying a threshold allows you to select or reject / discard a token, using zero or 0 to indicate rejection of a particular attention head and one or 1 to indicate selection of a particular attention head. For example, a threshold representing the 50th percentile (or median) of the row summation of the column can be used. In another example, thresholds located at different percentiles of the row summation of the column can be used. In yet another example, a threshold at the mean of the row summation of the column can be used. In yet another example, a fixed value of the threshold, empirically predetermined, can be used. In yet another example, the value of the threshold can be determined to optimize one or more metrics, such as representing the accuracy of the token. Using the threshold, the selection bits of the attention heads for a given token are determined. The selection bits can be rearranged to form a binary vector or binary feature vector for the given token. For example, for H attention heads, each token is represented by an 8-bit binary vector (instead of, for example, a 128-bit FP16 attention weight vector).

[0102] In an implementation where binary vectors represent token pages (rather than individual tokens), each binary vector can have a length of H × page size (where H is the number of attention heads in the converter-based neural network, and page size is the number of tokens per KV cache page). There can be P binary vectors, where P will be equal to the number of pages. The selected bits of the tokens for a page are concatenated together in an interleaved manner (e.g., cyclically) to form the binary vector. For example, assuming page P0 includes two tokens (page size = 2), T0 and T1, then the binary vector corresponding to token T0 with H = 4 heads is [0, 0, 0, 0], and the binary vector for token T1 is [1, 1, 1, 1]. Then, the binary vector for page P0 is [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1].

[0103] Using binary vectors as an alternative representation of tokens allows for low-overhead token pruning algorithms because the mathematical computation of binary vectors is far more efficient than that of floating-point vectors. Furthermore, low-overhead token pruning algorithms do not involve clustering; instead, they perform bucketing or grouping. While clustering-based techniques can produce accurate cluster centroids as surrogates, these techniques can suffer from high overhead due to the computationally intensive nature of clustering algorithms. Some studies have shown that a significant portion of the computational cycles is spent performing clustering while implementing token pruning.

[0104] To efficiently prune redundant or unrepresentative tokens and find representatives for pruned tokens, grouping or bucketing techniques can be implemented. Grouping or bucketing techniques can be based on low-overhead weak clustering / grouping algorithms. For example... Figure 10 As previously shown, the cache budget is divided into two parts: CIMPORTANT and C REPRESENTATIVE The first part selects the top C that will remain. IMPORTANT One portion of the tokens (e.g., important tokens) is used to store C tokens for discarding (e.g., unimportant tokens or tokens to be pruned). IMPORTANT Number of agents / representatives.

[0105] In some embodiments, appropriate techniques may be implemented to determine C. IMPORTANT The quantity is a significant token. An example of a suitable technique could include a normalized row summation based on attention weights (such as...). Figure 12 (as shown in 1202 previously) Select these C IMPORTANT Of the tokens, only the subset with the highest normalized row summation value is retained. In some embodiments, C is selected based on the attention weights of each token. IMPORTANT The attention mechanism assigns a token to each input token and compares its attention weight to a threshold to classify whether the token is important. Here, attention weight refers to a value indicating the relevance of the input token to the final output of the attention mechanism, the contribution of the input token to the attention output, or the probability that each token should receive attention. Relevance or contribution can be measured in terms of the attention weight corresponding to the token. The attention weight matrix can be calculated based on the query matrix and the key matrix. In some cases, the attention weight of a token can be determined using the distance between the token and the currently being processed token. In other cases, the attention score for each token can be calculated based on the attention weight matrix generated in the attention mechanism (e.g., an attention weight matrix generated based on the query matrix and the key matrix).

[0106] For the remaining less important tokens, KVCrush applies a weak clustering / grouping algorithm to represent them using surrogate or representative tokens, such as Figure 13 As shown, it illustrates a selection representative token according to some embodiments of this disclosure. Figure 9 The representative token selector 920 and / or token trimmer 918 can perform operations related to selecting representative tokens and trimming non-representative tokens.

[0107] The input to KVCrush can include S binary vectors, illustrated as 1206. A binary vector can be provided for each token. When KVCrush operates based on a KV cache, the input can include P binary vectors or P binary feature vectors, where P is the number of KV cache pages.

[0108] KVCrush processing involves selecting one or more anchor vectors, or one or more anchor points. An exemplary anchor vector 1302 is illustrated. The length of anchor vector 1302 matches the length of a binary vector. One or more anchor vectors serve as one or more points (or invariant points) in a multidimensional feature space, based on which distances to these points from other points in the multidimensional feature space can be calculated, and these distances can be used as proxy or heuristic strategies for grouping or bucketing similar points together.

[0109] In one example, the values ​​of the anchor vector are determined randomly (e.g., one / 1 or zero / 0 are randomly selected for each value of the anchor vector based on a binomial probability distribution). Random anchor vectors can include random vectors of zeros and ones. In one example, the values ​​of the anchor vector are determined based on the mean of the values ​​of a binary vector. Mean-based anchors can include a vector of the mean values ​​of binary vectors. In one example, the values ​​of the anchor vector are determined based on the median of the values ​​of a binary vector. Median-based anchors can include a vector of the median values ​​of binary vectors. In one example, the values ​​of the anchor vector have alternating zeros and ones. An anchor vector with alternating zeros and ones can include alternating sequences of zeros and ones, or ones and zeros. In one example, the values ​​of the anchor vector have a single pattern of zeros and ones.

[0110] KVCrush processing can use one or more anchor vectors 1302 to perform a grouping or bucketing process, which may include calculating one or more distances between one or more binary vectors representing one or more tokens and one or more anchor vectors. For a binary vector representing a token (e.g., the binary vector in the binary vectors illustrated in 1206), a distance to the anchor vector can be calculated relative to that binary vector. For S binary vectors, S distances can be calculated relative to the anchor vectors, one distance per token. For P binary vectors, P distances can be calculated relative to the anchor vectors, one distance per KV cache page. The calculated distances (e.g., S distances) are illustrated in 1304.

[0111] If more than one anchor vector is used, a composite distance can be calculated for the binary vectors, based on the distances between the binary vectors and the different anchor vectors. For S binary vectors, S composite distances can be calculated relative to multiple anchor vectors, one composite distance per token. For P binary vectors, P composite distances can be calculated relative to multiple anchor vectors, one composite distance per KV cache page.

[0112] In some cases, the distance between a binary vector and an anchor vector is called the Hamming distance. The Hamming distance is simple to calculate and measures the number of positions where the binary vector and anchor vector differ (or are the same). The Hamming distance can be calculated by counting the number of bit flips required to convert the binary vector to the anchor vector. The Hamming distance ranges from 0 (for identical vectors) to the length of the binary vector (the same as the anchor vector).

[0113] In some cases, the distance between a binary vector and an anchor vector is called the Jaccard distance. The Jaccard distance measures the difference between binary and anchor vectors by comparing their set properties. The Jaccard distance can be calculated as 1 - (intersection size / union size) of the binary and anchor vectors. The Jaccard distance can be useful when considering binary and anchor vectors as sets of binary features. The Jaccard distance ranges from 0 (identical sets) to 1 (completely different sets).

[0114] In some cases, the distance between a binary vector and an anchor vector is called the cosine distance. The cosine distance measures the cosine of the angle between the binary vector and the anchor vector. To measure the cosine distance between the binary vector and the anchor vector, the overlap of the "1" positions between the binary vector and the anchor vector is determined or measured. The cosine distance ranges from 0 (identical vectors) to 1 (orthogonal / dissimilar vectors).

[0115] In some cases, the distance between the binary vector and the anchor vector is the matching distance. The matching distance can be calculated as the proportion of matching positions between the binary vector and the anchor vector. The matching distance can count both zeros and ones in the matching, and can be used when both zeros and ones are equally important to the distance. The matching distance ranges from 0 (completely different) to 1 (completely identical).

[0116] In some cases, the distance between a binary vector and an anchor vector is represented by a dice coefficient. Similar to Jaccard distance, the dice coefficient assigns higher weight to positions that share one position. The dice coefficient can be calculated as 2 * (the number of shared ones in the binary and anchor vectors) / (all ones in both the binary and anchor vectors). The dice coefficient ranges from 0 (completely different) to 1 (completely identical).

[0117] In some cases, the composite distance between a binary vector and multiple anchor vectors is the Euclidean distance. The Euclidean distance can be calculated as a straight-line distance based on the distances between the binary vector and the anchor vectors. If the individual distances between the binary vector and the anchor vectors are e1, e2, and e3, then the Euclidean distance will be the square root of the sum of the squared distances, or... In some embodiments, for each binary vector, a separate distance (e.g., Hamming distance) can be computed for each anchor vector. The Hamming distance can form a distance vector, and the magnitude of the distance vector can be used as the composite distance of the binary vectors.

[0118] In some cases, the composite distance between a binary vector and multiple anchor vectors is the Manhattan distance, which can be measured as a grid. The Manhattan distance can be calculated as the sum of absolute distances. If the individual distances between the binary vector and the anchor vectors are e1, e2, and e3, then the Manhattan distance will be the sum of the absolute distances, or... .

[0119] In some cases, the composite distance between a binary vector and multiple anchor vectors is the Chebyshev distance, which is the maximum value of the measured distances. If the individual distances between the binary vector and the anchor vectors are e1, e2, and e3, then the Chebyshev distance will be the maximum value of the absolute distance, or... .

[0120] The grouping or bucketing process also includes setting buckets based on the cache budget. If the cache budget is allocated to store C in a KV cache... REPRESENTATIVE If a token is used as a representative or proxy for an unimportant token, then C is set. REPRESENTATIVE There are one bucket, illustrated as 1306. One or more distance ranges corresponding to one or more buckets can be determined. The distance ranges can be divided into C equal parts. REPRESENTATIVE A uniformly sized distance range, and each bucket can be assigned to C. REPRESENTATIVE The distance range is a range within a distance range. The distance range can be divided into C based on percentiles. REPRESENTATIVE A distance range, and each bucket can be assigned to C. REPRESENTATIVE Distance ranges within a distance range. The advantage of dividing distance ranges based on percentiles is that it ensures approximately the same number or percentage of tokens are allocated or bucketed. The distance ranges determined for each bucket can be the same size, but in some cases, these distance ranges can have different sizes.

[0121] One or more tokens can be assigned to one or more buckets based on one or more distance ranges and one or more distances. Distance (illustrated as 1304) can be placed into C. REPRESENTATIVE Within a bucket, illustrated as 1306. Placing a distance or token into a bucket involves checking if the distance falls within the specific distance range assigned to a given bucket. No sorting is required.

[0122] After bucketing or grouping the tokens using distance, a representative token can be selected from each bucket. For example, the representative token can be selected randomly or arbitrarily. In some cases, the first token assigned to a bucket is selected as the representative token. In other cases, the last token assigned to a bucket is selected as the representative token. The selected representative token... Figure 13 The text can have a checkmark; unselected tokens are in... Figure 13 X can be present in C. As a result, from C... REPRESENTATIVE Choose C from the number of buckets REPRESENTATIVE A number of representative tokens. The selected representative token can be used as a representative / proxy for that representative token, or for one or more other tokens assigned to a given bucket. The KV cache (computed key and value tensors) corresponding to the representative token is used as the representative / proxy key and value tensors for all tokens assigned to a given bucket. Other tokens (e.g., KV caches corresponding to other tokens) are discarded or evicted. These C REPRESENTATIVE Each representative token and the previously selected C IMPORTANT Each token is ultimately retained in the KV cache.

[0123] Using one or more anchors to find representatives / proxies in a one-time bucketing approach allows KVCrush to perform pruning with only S (or P) distance comparisons, instead of the O(S) operations performed by clustering algorithms such as k-means clustering. 2 (or O(P)) 2 Distance calculation.

[0124] Choosing appropriate anchor vectors can affect the accuracy of delegates or proxies. In one implementation, KVCrush uses anchor vectors (binary vectors of length equal to the number of heads) within the same binary space to form token buckets based on the Hamming distance between the token and that anchor vector. Using different types of anchors can affect the accuracy of the buckets relative to ideal clustering formed using the k-means algorithm. Using more anchors can improve the accuracy of the buckets with minimal additional overhead when calculating the final distance.

[0125] Note that the k-means clustering algorithm exhibits significant inefficiency in KV cache compression due to two main factors. First, the large size of the input vectors presents a challenge. For example, in a converter-based model with 8 billion parameters, each input key (or value) vector includes 128 FP16 values. By employing KVCrush's efficient representation of key-value states, a binary vector of length 32 (corresponding to the number of attention heads) can be generated to represent each token (key-value pair). This method not only reduces the data size by 64 times but also achieves faster distance computation using Hamming distance. Second, the time complexity associated with the clustering algorithm used for data grouping is significant. KVCrush's low-overhead pruning algorithm completes the selection of values ​​to be retained in the cache (selecting representatives / proxy members) in linear time, contrasting sharply with the quadratic time complexity of k-means clustering.

[0126] Using KVCrush at the KV cache page level The techniques for KVCrush described herein can be implemented at the KV cache page level rather than the token level. At the page level, a KV cache page has multiple tokens. To evaluate KVCrush in a KV cache pagination implementation, row-by-row attention weights at the page level (each page is represented as a row of attention weights) can be used to evict unimportant pages and identify important pages. Furthermore, the binary vector of the page (e.g., using...) Figures 11-12 The operation computation shown can be formed by concatenating the binary vectors of all tokens within the page in an interleaved manner. Furthermore, distances can be calculated relative to one or more anchor vectors for the page's binary vector. These distances can be used to bucket the page. Multiple buckets are set up based on the cache budget allocated to the representative KV cache for storing the page. Representative pages can be selected (e.g., randomly) from a given bucket, and the KV cache computed for the tokens of the representative page can be used as a representative or proxy for the KV cache of the page allocated to the given bucket.

[0127] Methods for KV cache management Figure 14 This is a flowchart illustrating a method for KV cache management according to some embodiments of the present disclosure. Method 1400 can be performed by... Figure 9 This is implemented using one or more components of the KV cache controller 912. Method 1400 can be implemented to provide efficient token pruning. Method 1400 can be executed using a computing device, for example, Figure 16 The computing device in the middle is 1600.

[0128] In 1402, one or more binary vectors are computed based on one or more attention weight matrices calculated by one or more attention heads of the neural network. These one or more binary vectors may represent one or more tokens requesting the neural network.

[0129] In 1404, one or more tokens are assigned to one or more buckets based on one or more distances between one or more binary vectors and anchor vectors.

[0130] In 1406, a representative token is selected from one or more tokens assigned to one or more buckets.

[0131] In 1408, the representative key tensor and representative value tensor, computed by one or more attention heads of the neural network for the representative token, are stored in a key-value cache.

[0132] In 1410, the token is determined to be pruned based on the importance of one or more tokens. The token is then assigned to the bucket.

[0133] In 1412, the representative key tensor and the representative value tensor are provided from the key-value cache to the computational logic that is performing one or more operations on one or more attention heads.

[0134] Figure 15 This is a flowchart illustrating a method for KV cache management according to some embodiments of the present disclosure. Method 1500 can be performed by... Figure 9 This is implemented using one or more components of the KV cache controller 912. Method 1500 can be implemented to provide efficient token pruning. Method 1500 can be executed using a computing device, for example, Figure 16 The computing device in the middle is 1600.

[0135] In 1502, one or more binary vectors representing one or more tokens that represent a request to a neural network can be computed.

[0136] In 1504, one or more selected tokens from one or more tokens can be assigned to a bucket based on one or more binary vectors (e.g., one or more binary vectors representing one or more selected tokens).

[0137] In 1506, you can select a representative token to represent one or more selected tokens assigned to the bucket.

[0138] In version 1508, representative key tensors and representative value tensors computed for representative tokens can be stored in a key-value cache. Representative key and value tensors serve as proxies or representatives for key tensors and value tensors computed for one or more selected tokens assigned to that bucket. Key tensors and value tensors computed for one or more non-representative tokens are discarded or evicted. Representative key and value tensors computed for representative tokens are retained.

[0139] In 1510, representative key tensors and representative value tensors are provided to the operations of a neural network (e.g., attention heads) that operate on one or more selected tokens assigned to the bucket.

[0140] Exemplary computing device Figure 16 This is a block diagram of an apparatus or system (e.g., an exemplary computing device 1600) according to some embodiments of the present disclosure. One or more computing devices 1600 may be used to implement the functions described herein in conjunction with the accompanying drawings. Figure 16 The various components shown may be included in the computing device 1600, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in the computing device 1600 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. Furthermore, in various embodiments, the computing device 1600 may not include... Figure 16 The computing device 1600 may include one or more components as shown, and may include interface circuitry for coupling to said one or more components. For example, the computing device 1600 may not include a display device 1606, but may include display device interface circuitry (e.g., connector and driver circuitry) to which the display device 1606 may be coupled. In another set of examples, the computing device 1600 may not include an audio input device 1618 or an audio output device 1608, but may include audio input or output device interface circuitry (e.g., connector and support circuitry) to which the audio input device 1618 or the audio output device 1608 may be coupled.

[0141] Computing device 1600 may include processing device 1602 (e.g., one or more processing devices, one or more processing devices of the same type, or one or more processing devices of different types). Processing device 1602 may include electronic circuitry that processes electronic data from data storage elements (e.g., registers, memories, resistors, capacitors, qubit units) to transform the electronic data into other electronic data that can be stored in registers and / or memories. Examples of processing device 1602 may include CPUs, GPUs, quantum processors, machine learning processors, artificial intelligence processors, neural network processors, neural processing units (NPUs), artificial intelligence accelerators, application-specific integrated circuits (ASICs), analog signal processors, analog computers, microprocessors, digital signal processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), data processing units (DPUs), etc.

[0142] Computing device 1600 may include memory 1604, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. Memory 1604 includes one or more non-transitory computer-readable storage media. In some embodiments, memory 1604 may include memory sharing a die with processing device 1602.

[0143] In some embodiments, memory 1604 includes one or more non-transitory computer-readable media storing instructions executable to perform the operations described herein in conjunction with the accompanying drawings, such as the methods and operations illustrated in the drawings. In some embodiments, memory 1604 includes one or more non-transitory computer-readable media storing instructions executable to perform... Figures 10-13 One or more operations are illustrated. In some embodiments, memory 1604 includes one or more non-transitory computer-readable media storing instructions that are executable to perform... Figure 14 The operation of method 1400 is described. Exemplary components that can be encoded as instructions and stored in memory 1604 are depicted. Memory 1604 may store instructions encoding one or more exemplary components (e.g., one or more components of KV cache controller 912). Instructions stored in one or more non-transitory computer-readable media may be executed by processing device 1602.

[0144] In some embodiments, memory 1604 may store data, such as data structures, binary data, bits, metadata, files, binary large objects, etc., as described herein in conjunction with the accompanying drawings. For example, memory 1604 may include a KV cache 904, and the KV cache 904 may store key tensors and value tensors. Memory 1604 may store representative key tensors and representative value tensors. Memory 1604 may store the importance score of a token. Memory 1604 may store a binary vector representing a token, or a KV cache page. Memory 1604 may store the association between a token and a bucket. Memory 1604 may store the input data, intermediate data, and output data of the KVCrush algorithm.

[0145] In some embodiments, computing device 1600 may include communication device 1612 (e.g., one or more communication devices). For example, communication device 1612 may be configured to manage wired and / or wireless communications for transmitting data to and from computing device 1600. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can transmit data through a non-solid-state medium using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wiring, but in some embodiments they may not. Communication device 1612 may implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as “3GPP2”), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks. This abbreviation stands for Global Microwave Access, and it is a product certification mark indicating compliance and interoperability with the IEEE 802.16 standard. Communication equipment 1612 can operate according to: Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication equipment 1612 can operate according to: GSM Evolution Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication equipment 1612 can operate according to: Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, and above. In other embodiments, the communication device 1612 may operate according to other wireless protocols. The computing device 1600 may include an antenna 1622 to facilitate wireless communication and / or receive other wireless communications (e.g., radio frequency transmissions). The computing device 1600 may include receiver circuitry and / or transmitter circuitry. In some embodiments, the communication device 1612 may manage wired communication, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication device 1612 may include multiple communication chips.For example, the first communication device 1612 may be dedicated to shorter-range wireless communication, such as Wi-Fi or Bluetooth, and the second communication device 1612 may be dedicated to longer-range wireless communication, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication device 1612 may be dedicated to wireless communication, and the second communication device 1612 may be dedicated to wired communication.

[0146] The computing device 1600 may include a power supply / power circuit 1614. The power supply / power circuit 1614 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1600 to an energy source (e.g., DC power, AC power, etc.) separate from the computing device 1600.

[0147] The computing device 1600 may include a display device 1606 (or corresponding interface circuitry, as described above). The display device 1606 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0148] The computing device 1600 may include an audio output device 1608 (or a corresponding interface circuit, as described above). The audio output device 1608 may include any device that generates auditory indications, such as a speaker, headphones, or earphones.

[0149] The computing device 1600 may include an audio input device 1618 (or a corresponding interface circuit, as described above). The audio input device 1618 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).

[0150] The computing device 1600 may include a GPS device 1616 (or a corresponding interface circuit, as described above). The GPS device 1616 may communicate with a satellite-based system and may receive the location of the computing device 1600, as is known in the art.

[0151] The computing device 1600 may include a sensor 1630 (or one or more sensors). The computing device 1600 may include corresponding interface circuitry, as described above. The sensor 1630 can sense physical phenomena and convert them into electrical signals that can be processed by, for example, the processing device 1602. Examples of the sensor 1630 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, strain sensors, humidity sensors, distance sensors, range sensors, time-of-flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0152] The computing device 1600 may include another output device 1610 (or a corresponding interface circuit, as described above). Examples of the other output device 1610 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a haptic output device, a gas output device, a vibration output device, a lighting output device, a home automation controller, or an additional storage device.

[0153] The computing device 1600 may include another input device 1620 (or a corresponding interface circuit, as described above). Examples of the other input device 1620 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a quick-response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0154] The computing device 1600 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, personal digital assistant (PDA), personal computer, remote control, wearable device, headwear, glasses, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, Internet of Things device, or wearable computing system. In some embodiments, the computing device 1600 may be any other electronic device that processes data.

[0155] Select Example Example 1 provides an apparatus comprising: machine-readable instructions; one or more memories storing a key-value cache; and at least one computer processor that, when executing the machine-readable instructions, performs the following operations: computes one or more binary vectors of one or more tokens representing a request for a neural network; assigns one or more selected tokens from the one or more tokens to a bucket based on the one or more binary vectors; selects a representative token to represent the one or more selected tokens assigned to the bucket; stores a representative key tensor and a representative value tensor computed for the representative tokens in the key-value cache; and provides the representative key tensor and the representative value tensor from the key-value cache to the operation of the neural network operating on the one or more selected tokens assigned to the bucket.

[0156] Example 2 provides the apparatus of Example 1, wherein computing one or more binary vectors comprises: computing one or more binary vectors based on one or more attention weight matrices computed by one or more attention heads of a neural network.

[0157] Example 3 provides an apparatus for Example 1 or 2, wherein computing one or more binary vectors includes: computing a normalized row summation of an attention weight matrix computed by a neural network; and binarizing the normalized row summation using a threshold.

[0158] Example 4 provides an apparatus for any of Examples 1-3, wherein one or more binary vectors comprise binary values ​​for each of one or more attention heads in a neural network.

[0159] Example 5 provides the apparatus of Example 4, where the binary value is a selection bit that indicates whether a given attention head in the neural network focuses on or ignores a particular token.

[0160] Example 6 provides an apparatus for any of Examples 1-5, wherein assigning one or more selected tokens to a bucket comprises: calculating one or more distances between one or more binary vectors representing one or more tokens and anchor vectors; determining a distance range corresponding to the bucket; and assigning one or more selected tokens to the bucket based on the distance range and one or more distances.

[0161] Example 7 provides an apparatus of any of Examples 1-6, wherein assigning one or more selected tokens to a bucket comprises: randomly determining one or more values ​​of an anchor vector; and assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and the anchor vector.

[0162] Example 8 provides an apparatus of any of Examples 1-6, wherein assigning one or more selected tokens to a bucket comprises: determining one or more values ​​of an anchor vector based on the mean of one or more binary vectors; and assigning one or more selected tokens to the bucket based on one or more distances between the one or more binary vectors and the anchor vector.

[0163] Example 9 provides an apparatus of any of Examples 1-6, wherein assigning one or more selected tokens to a bucket comprises: assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and an anchor vector having alternating zeros and ones.

[0164] Example 10 provides an apparatus for any of Examples 6-9, wherein one or more distances include one or more Hamming distances.

[0165] Example 11 provides an apparatus of any of Examples 1-10, wherein selecting a representative token from one or more tokens selected from a bucket includes randomly selecting a representative token from one or more selected tokens.

[0166] Example 12 provides an apparatus of any of Examples 1-11, wherein: the representative key tensor and the representative value tensor are computed by the attention head of the neural network for the representative tokens; and the representative key tensor and the representative value tensor represent one or more key tensors and one or more value tensors computed by the attention head of the neural network for one or more selected tokens.

[0167] Example 13 provides an apparatus of any one of Examples 1-12, wherein at least one computer processor further performs the following operations: assigning one or more further selected tokens from one or more tokens to another bucket based on one or more binary vectors; selecting another representative token to represent the one or more further selected tokens assigned to the other bucket; storing another representative key tensor and another representative value tensor computed for the other representative token in a key-value cache; determining that the requested other token is one of the one or more further selected tokens assigned to the other bucket; and operations of a neural network that provide the other representative key tensor and another representative value tensor from the key-value cache for operation on the other token.

[0168] Example 14 provides an apparatus of any of Examples 1-13, wherein at least one computer processor further performs the following operations: determining that one or more selected tokens will be pruned based on one or more importance values ​​of the one or more selected tokens.

[0169] Example 15 provides an apparatus for any of Examples 1-13, wherein at least one computer processor further performs the following operations: evicting one or more key tensors and one or more value tensors computed against one or more remaining tokens assigned to the bucket that were not selected as representative tokens.

[0170] Example 16 provides one or more non-transitory computer-readable media storing instructions executable by a processor to perform operations for managing a key-value cache, the operations including: computing one or more binary vectors of one or more tokens representing a request for a neural network; assigning one or more selected tokens from the one or more tokens to a bucket based on the one or more binary vectors; selecting a representative token to represent the one or more selected tokens assigned to the bucket; storing a representative key tensor and a representative value tensor computed for the representative tokens in the key-value cache; and providing the representative key tensor and the representative value tensor from the key-value cache to a neural network operating on the one or more selected tokens assigned to the bucket.

[0171] Example 17 provides one or more non-transitory computer-readable media of Example 16, wherein computing one or more binary vectors includes: computing one or more binary vectors based on one or more attention weight matrices computed by one or more attention heads of a neural network.

[0172] Example 18 provides one or more non-transitory computer-readable media of Example 16 or 17, wherein computing one or more binary vectors includes: computing a normalized row summation of an attention weight matrix computed by a neural network; and binarizing the normalized row summation using a threshold.

[0173] Example 19 provides one or more non-transitory computer-readable media of any of Examples 16-18, wherein one or more binary vectors include binary values ​​for each of one or more attention heads in a neural network.

[0174] Example 20 provides one or more non-transitory computer-readable media of Example 19, wherein binary values ​​are selection bits that indicate whether a given attention head in the attention head of a neural network pays attention to a particular token or ignores a particular token.

[0175] Example 21 provides one or more non-transitory computer-readable media of any of Examples 16-20, wherein assigning one or more selected tokens to a bucket comprises: calculating one or more distances between one or more binary vectors representing one or more tokens and anchor vectors; determining a distance range corresponding to the bucket; and assigning one or more selected tokens to the bucket based on the distance range and one or more distances.

[0176] Example 22 provides one or more non-transitory computer-readable media of any of Examples 16-21, wherein assigning one or more selected tokens to a bucket comprises: randomly determining one or more values ​​of an anchor vector; and assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and the anchor vectors.

[0177] Example 23 provides one or more non-transitory computer-readable media of any of Examples 16-21, wherein assigning one or more selected tokens to a bucket comprises: determining one or more values ​​of an anchor vector based on the mean of one or more binary vectors; and assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and the anchor vector.

[0178] Example 24 provides one or more non-transitory computer-readable media of any of Examples 16-21, wherein assigning one or more selected tokens to a bucket comprises: assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and anchor vectors having alternating zeros and ones.

[0179] Example 25 provides one or more non-transitory computer-readable media of any of Examples 21-24, wherein one or more distances include one or more Hamming distances.

[0180] Example 26 provides one or more non-transitory computer-readable media of any of Examples 16-25, wherein selecting a representative token from one or more tokens allocated to a bucket includes randomly selecting a representative token from one or more tokens.

[0181] Example 27 provides one or more non-transitory computer-readable media of any of Examples 16-26, wherein: the representative key tensor and the representative value tensor are computed by the attention head of a neural network for representative tokens; and the representative key tensor and the representative value tensor represent one or more key tensors and one or more value tensors computed by the attention head of the neural network for one or more selected tokens.

[0182] Example 28 provides one or more non-transitory computer-readable media of any of Examples 16-27, wherein the processor further performs the following operations: assigning one or more further selected tokens from one or more tokens to another bucket based on one or more binary vectors; selecting another representative token to represent the one or more further selected tokens assigned to the other bucket; storing another representative key tensor and another representative value tensor computed for the other representative token in a key-value cache; determining that the requested other token is one of the one or more further selected tokens assigned to the other bucket; and operations of a neural network that provide the other representative key tensor and another representative value tensor from the key-value cache for operation on the other token.

[0183] Example 29 provides one or more non-transitory computer-readable media of any of Examples 16-28, wherein the processor further performs the following operations: determining that one or more selected tokens will be pruned based on one or more importance values ​​of the one or more selected tokens.

[0184] Example 30 provides one or more non-transitory computer-readable media of any of Examples 16-29, wherein the processor also performs the following operations: evicts one or more key tensors and one or more value tensors computed against one or more remaining tokens assigned to the bucket that were not selected as representative tokens from the key-value cache.

[0185] Example 31 provides a method for managing a key-value cache, comprising: computing one or more binary vectors of one or more tokens representing a request for a neural network; assigning one or more selected tokens from the one or more tokens to a bucket based on the one or more binary vectors; selecting a representative token to represent the one or more selected tokens assigned to the bucket; storing a representative key tensor and a representative value tensor computed for the representative tokens in the key-value cache; and providing the representative key tensor and the representative value tensor from the key-value cache to operations of a neural network operating on the one or more selected tokens assigned to the bucket.

[0186] Example 32 provides the method of Example 31, wherein computing one or more binary vectors includes: computing one or more binary vectors based on one or more attention weight matrices computed by one or more attention heads of a neural network.

[0187] Example 33 provides a method from Example 31 or 32, wherein computing one or more binary vectors includes: computing a normalized row summation of the attention weight matrix computed by the neural network; and binarizing the normalized row summation using a threshold.

[0188] Example 34 provides a method for any of Examples 31-33, wherein one or more binary vectors comprise binary values ​​for each of one or more attention heads in a neural network.

[0189] Example 35 provides the method of Example 34, where the binary value is a selection bit that indicates whether a given attention head in the neural network focuses on or ignores a particular token.

[0190] Example 36 provides a method of any one of Examples 31-35, wherein assigning one or more selected tokens to a bucket comprises: calculating one or more distances between one or more binary vectors representing one or more tokens and anchor vectors; determining a distance range corresponding to the bucket; and assigning one or more selected tokens to the bucket based on the distance range and one or more distances.

[0191] Example 37 provides a method of any of Examples 31-36, wherein assigning one or more selected tokens to a bucket includes: randomly determining one or more values ​​of an anchor vector; and assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and the anchor vector.

[0192] Example 38 provides a method of any of Examples 31-36, wherein assigning one or more selected tokens to a bucket comprises: determining one or more values ​​of an anchor vector based on the mean of one or more binary vectors; and assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and the anchor vector.

[0193] Example 39 provides a method of any of Examples 31-36, wherein assigning one or more selected tokens to a bucket comprises: assigning one or more selected tokens to the bucket based on one or more distances between one or more binary vectors and an anchor vector having alternating zeros and ones.

[0194] Example 40 provides a method for any of Examples 36-39, where one or more distances include one or more Hamming distances.

[0195] Example 41 provides a method of any of Examples 31-40, wherein selecting a representative token from one or more selected tokens assigned to a bucket includes randomly selecting a representative token from one or more selected tokens.

[0196] Example 42 provides a method for any of Examples 31-41, wherein: the representative key tensor and the representative value tensor are computed by the attention head of the neural network for the representative tokens; and the representative key tensor and the representative value tensor represent one or more key tensors and one or more value tensors computed by the attention head of the neural network for one or more selected tokens.

[0197] Example 43 provides a method of any one of Examples 31-42, further comprising: assigning one or more further selected tokens from one or more tokens to another bucket based on one or more binary vectors; selecting another representative token to represent the one or more further selected tokens assigned to the other bucket; storing another representative key tensor and another representative value tensor computed for the other representative token in a key-value cache; determining that the requested other token is one of the one or more further selected tokens assigned to the other bucket; and operations of a neural network that provide the other representative key tensor and another representative value tensor from the key-value cache for operation on the other token.

[0198] Example 44 provides a method for any of Examples 31-43, further comprising: determining that one or more selected tokens will be pruned based on one or more importance values ​​of the one or more selected tokens.

[0199] Example 45 provides a method for any of Examples 31-44, further comprising: evicting one or more key tensors and one or more value tensors computed against one or more remaining tokens assigned to the bucket that were not selected as representative tokens from the key-value cache.

[0200] Example A includes a device that includes means for performing any of the methods in Examples 31-45.

[0201] Example B includes a KV cache controller as described in this article.

[0202] Example C includes a computing system having a computing core, a key-value cache, and a key-value cache controller as described herein (e.g., Figure 9 ).

[0203] Variations and other notes Although the operations of the example methods shown and described with reference to the accompanying drawings are illustrated as being performed once each in a specific order, it will be understood that these operations can be performed in any suitable order and repeated as needed. Furthermore, one or more operations can be performed in parallel. Additionally, the operations shown in the drawings can be combined, or may include more or fewer details than described.

[0204] The various implementation methods described herein may relate to artificial intelligence, machine learning, and deep learning. Deep learning may be a subset of machine learning. Machine learning may be a subset of artificial intelligence. When referring to deep learning models, machine learning models may be used alternatively if applicable to a particular application. When referring to deep learning models, digital signal processing systems may be used alternatively if applicable to a particular application.

[0205] The foregoing description of the illustrated embodiments of this disclosure, including those described in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise forms disclosed. While specific embodiments and examples of this disclosure have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to this disclosure in accordance with the detailed description above.

[0206] For illustrative purposes, specific figures, materials, and configurations are described to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or with only some of the described aspects. In other instances, well-known features have been omitted or simplified to avoid obscuring the illustrative embodiments.

[0207] Furthermore, reference has been made to the accompanying drawings, which form part of this document, illustrating possible embodiments. It will be understood that other embodiments may be utilized, and structural or logical changes may be made, without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0208] Various operations can be described sequentially as a plurality of discrete actions or operations in a manner most helpful for understanding the disclosed subject matter. However, the order of description should not be construed as implying that these operations are necessarily order-dependent. Specifically, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed or operations may be omitted from the description.

[0209] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When referring to a measurement range, the term "between" includes both ends of the measurement range.

[0210] In the following detailed description, various aspects of the illustrative embodiments will be described using terminology commonly used by those skilled in the art to convey the essence of their work to others skilled in the art.

[0211] The terms “substantially,” “near,” “roughly,” “close to,” and “about” generally refer to a target value as described herein or known in the art within + / - 20%. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to a target value as described herein or known in the art within + / - 5-20%.

[0212] Furthermore, the terms "comprising," "including," "having," or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to the method, process, or apparatus. Additionally, the term "or" refers to inclusive "or," not exclusive "or."

[0213] The systems, methods, and apparatuses disclosed herein each have several innovative aspects, and no single aspect is the sole reason for all the desirable properties disclosed herein. Details of one or more embodiments of the subject matter described herein are set forth in the specification and accompanying drawings.

Claims

1. A method for managing a key-value cache, comprising: Calculate one or more binary vectors that represent one or more tokens for a request to the neural network; Based on the one or more binary vectors, assign one or more selected tokens from the one or more tokens to the bucket; Select a representative token to represent the one or more selected tokens assigned to the bucket; The key-value cache stores the representative key tensor and the representative value tensor computed for the representative token; as well as From the key-value cache, the representative key tensor and the representative value tensor are provided to the neural network that operates on the one or more selected tokens assigned to the bucket.

2. The method according to claim 1, wherein, Calculating the one or more binary vectors includes: calculating the one or more binary vectors based on one or more attention weight matrices calculated by one or more attention heads of the neural network.

3. The method according to claim 1 or 2, wherein, Calculating the one or more binary vectors includes: Calculate the normalized row summation of the attention weight matrix computed by the neural network; and The sum of the normalized rows is binarized using a threshold.

4. The method according to claim 1 or 2, wherein, The binary vectors in the one or more binary vectors include: the binary value of each attention head in one or more attention heads of the neural network.

5. The method according to claim 4, wherein, The binary value is a selection bit, indicating whether a given attention head in the neural network focuses on or ignores a particular token.

6. The method according to claim 1 or 2, wherein, Assigning the one or more selected tokens to the bucket includes: Calculate the distance between the one or more binary vectors representing the one or more tokens and the anchor vectors; Determine the distance range corresponding to the bucket; and The selected tokens are assigned to the bucket based on the distance range and the one or more distances.

7. The method according to claim 1 or 2, wherein, Assigning the one or more selected tokens to the bucket includes: Randomly determine one or more values ​​of the anchor vector; and The one or more selected tokens are assigned to the bucket based on one or more distances between the one or more binary vectors and the anchor vector.

8. The method according to claim 1 or 2, wherein, Assigning the one or more selected tokens to the bucket includes: Based on the mean of the one or more binary vectors, determine one or more values ​​of the anchor vector; and The one or more selected tokens are assigned to the bucket based on one or more distances between the one or more binary vectors and the anchor vector.

9. The method according to any one of claims 1-6, wherein, Assigning the one or more selected tokens to the bucket includes: The one or more selected tokens are assigned to the bucket based on one or more distances between the one or more binary vectors and anchor vectors, wherein the anchor vectors have alternating zeros and ones.

10. The method according to claim 6, wherein, The one or more distances include one or more Hamming distances.

11. The method according to claim 1 or 2, wherein, Selecting the representative token from the one or more selected tokens assigned to the bucket includes: The representative token is randomly selected from the one or more selected tokens.

12. The method according to claim 1 or 2, wherein: The representative key tensor and the representative value tensor are computed by the attention head of the neural network for the representative token; and The representative key tensor and the representative value tensor represent one or more key tensors and one or more value tensors computed by the attention head of the neural network for the one or more selected tokens.

13. The method according to claim 1 or 2, further comprising: Based on the one or more binary vectors, assign one or more further selected tokens from the one or more tokens to another bucket; Select another representative token to represent one or more further selected tokens assigned to the other bucket; The key-value cache stores another representative key tensor and another representative value tensor computed for the other representative token. Determine that the other token requested is one of the one or more further selected tokens assigned to the other bucket; as well as The neural network operates by providing the other representative key tensor and the other representative value tensor from the key-value cache for operation on the other token.

14. The method according to claim 1 or 2, further comprising: The selected tokens will be pruned based on one or more importance values ​​of the selected tokens.

15. The method according to claim 1 or 2, further comprising: Evict one or more key tensors and one or more value tensors computed for the following tokens from the key-value cache: one or more other tokens assigned to the bucket that were not selected as the representative token.

16. One or more non-transitory computer-readable media storing instructions executable by a processor to perform the method according to any one of claims 1-15.

17. An apparatus comprising: Machine-readable instructions; One or more memories are used to store key-value caches; as well as At least one computer processor, which executes the method according to any one of claims 1-15 when executing the machine-readable instructions.

Images