A multi-granularity clustering-based scheme for key-value cache compression

Abstract

The key-value (LV) cache in this application accelerates inference in large language models (LLMs) by allowing attention operations to scale linearly, rather than quadratically, with the total sequence length. Since context lengths are long in modern LLMs, the KV cache size may exceed the model size, potentially negatively impacting throughput. To address this issue, a multi-granularity clustering-based scheme for KV cache compression is implemented. Clusters created at different clustering levels with variable precision approximate the key and value tensors corresponding to less important terms. Precision loss is reduced by using proxies generated at finer-grained clustering levels of subsets of more salient attention heads. More salient attention heads have a greater impact on model accuracy than less salient attention heads. When the impact on accuracy is low, latency is improved by retrieving proxies from a subset of less salient attention heads from faster memory.

Description

Background Technology

[0001] Deep neural networks (DNNs) are widely used in a variety of artificial intelligence applications, from computer vision to speech recognition and natural language processing, due to their ability to achieve high accuracy. However, this high accuracy comes at the cost of enormous computational costs. DNNs have extremely high computational demands due to the need to read and write large amounts of data and perform numerous operations. Attached Figure Description

[0002] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals refer to similar structural elements. In the accompanying drawings, embodiments are shown by way of example rather than by way of limitation.

[0003] Figure 1 An exemplary large language model is shown as an implementation of a transformer-based neural network according to some embodiments of the present disclosure.

[0004] Figure 2 A series converter module according to some embodiments of the present disclosure is shown.

[0005] Figure 3 Parallel converter modules according to some embodiments of the present disclosure are shown.

[0006] Figure 4 An attention layer of a converter module according to some embodiments of this disclosure is shown.

[0007] Figure 5 The computation in a self-attention layer without key-value (KV) caching is illustrated according to some embodiments of the present disclosure.

[0008] Figure 6 The computation in a self-attention layer with KV buffering according to some embodiments of the present disclosure is illustrated.

[0009] Figure 7 A multi-level memory system according to some embodiments of the present disclosure is shown.

[0010] Figure 8 A multi-level memory system with KV cache compression according to some embodiments of the present disclosure is shown.

[0011] Figure 9 Different clustering levels are shown according to some embodiments of this disclosure.

[0012] Figure 10 This is a flowchart illustrating the use of a proxy for KV cache retention and approximation according to some embodiments of this disclosure.

[0013] Figure 11This is a flowchart illustrating the determination of the salience of the attention head according to some embodiments of the present disclosure.

[0014] Figure 12 This is a flowchart illustrating an approximate KV caching method based on multi-granularity clustering that utilizes KV caches associated with unimportant terms, according to some embodiments of this disclosure.

[0015] Figure 13 This is a block diagram of an exemplary computing device according to some embodiments of the present disclosure. Detailed Implementation Overview

[0016] The past decade has witnessed the rapid development of data processing based on artificial intelligence (AI), especially deep neural networks (DNNs). Primarily due to the superior accuracy achieved by DNNs compared to human levels, they are widely used in computer vision, speech recognition, image processing, and video processing. Typically, a DNN consists of a sequence of layers. A DNN layer can include one or more deep learning operations (also known as "neural network operations"), such as convolution, matrix multiplication, layer normalization, batch normalization, SoftMax, pooling, element-wise operations, linear operations, and non-linear operations. While DNNs are efficient in analysis and prediction, they come at the cost of enormous computational power. DNNs can consume considerable power and runtime during training and inference.

[0017] Transformer-based neural networks, or transformer-based models, are a type of DNN that can be used to drive large language models (LLMs) and computer vision models (referred to as ViTs in the literature). Transformer-based neural networks are used in services and applications such as natural language processing, speech processing, general AI assistance, image captioning, object detection, video understanding, recommender systems, bioinformatics, time series prediction, reinforcement learning, and generative models to generate text, images, or music. Cloud companies can provide transformer-based neural networks as a managed service, where the neural network can be served by multiple distributed graphics processing unit (GPU) worker nodes, and the managed service can provide various services to multiple users based on their needs.

[0018] For some LLM or other machine learning models, an autoregressive transformer-based neural network is used. A transformer-based neural network generates one term at a time (e.g., one word at a time) based on input cues and the previous sequence of output terms generated by the transformer-based neural network so far. The process involving all operations performed in the transformer-based neural network is repeated term by term until the transformer-based neural network outputs a terminating term. When generating one term at a time, a key-value (KV) cache is introduced to avoid redundant computation. Specifically, the KV cache allows the reuse of key tensors and value tensors (attention outputs of operations in the transformer-based neural network) cached from previous terms. The KV cache stores the key and value tensors pre-computed in the attention computation and allows the reuse of these pre-computed key and value tensors when generating new terms.

[0019] The cached key and value tensors can include (intermediate) key and value tensors generated in an attention mechanism (e.g., one or more attention layers in a transformer-based neural network) during the generation of prior output lexics of a request. Here, a request refers to an instruction from a transformer-based neural network to generate one or more output lexics based on one or more input lexics. A request can include a request from a transformer-based neural network to generate one or more responses with one or more output lexics from input prompts having one or more input lexics. This generation may involve autoregressive generation of lexics, i.e., generating the next lexic involves using already generated lexics as part of the input lexics. A request can include or relate to one or more lexics. The cached key and value tensors can correspond to one or more lexics. Using a KV cache to store the cached key and value tensors can significantly reduce computation time and memory utilization. Intermediate key and value tensors can include key and value tensors generated across layers and across attention heads within a layer during lexic generation.

[0020] Here, the input or output data of deep learning operations, such as the attention output or intermediate attention output of the attention mechanism in the attention layer, can be arranged in a data structure called a tensor. A tensor is a data structure with multiple elements across one or more dimensions. Examples of tensors include vectors (one-dimensional (1D) tensors), matrices (two-dimensional (2D) tensors), three-dimensional (3D) tensors, four-dimensional (4D) tensors, and tensors with even higher dimensions. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimensions can be measured by multiple data points along an axis. The dimensions of a tensor can define the shape of the tensor. Attention mechanisms can generate attention outputs, such as key tensors and value tensors corresponding to one or more terms that can be cached in a KV cache to avoid redundant computation.

[0021] Key-value (KV) caching can accelerate LLM inference by allowing attention operations to scale linearly, rather than quadratically, with the total sequence length. A significant challenge for executing these transformer-based neural networks and serving multiple requests to them is managing the KV cache. Due to the large context lengths in modern LLMs, the KV cache size can exceed the model size, negatively impacting throughput. Effective use of the KV cache can reduce the cost of serving various requests, improve the throughput of managed services, and enhance the availability of managed services. Running neural networks with the limited memory budget of a KN cache (as is the case in most practical implementations) can be challenging. Because the KV cache size grows linearly with the sequence length (and each request can be enormous), managing the KV cache is not trivial. In some cases, the KV cache may require several times more memory than the memory used to store model parameters.

[0022] Some approaches address this challenge by minimizing the memory footprint of the KV cache through techniques such as discarding / skipping low-attention terms, quantization, and matrix approximation. One set of approaches retains terms based on their importance. In other words, important terms or a subset of terms are retained in the KV cache. Importance refers to the term's significance to the attention mechanism of the transformer-based neural network, or its contribution to the attention mechanism. In some cases, importance can be determined based on attention weights or distance from the current term. Here, when referring to term retention, it means that the KV cache corresponding to the term (with the cached key and value tensors) is retained and stored in the KV cache. When referring to discarding / removing / evicting terms, it means that the KV cache corresponding to the term (with the cached key and value tensors) is not stored or is retained in the KV cache. In another set of approaches, in addition to retaining important terms, methods focus on retaining less important terms by introducing noise, applying matrix approximation, implementing mixed-precision quantization, etc. These groups' solutions involve KV cache compression to reduce the size of the KV cache. If not managed properly, compression techniques can often suffer from accuracy loss.

[0023] To address this issue, a multi-granularity clustering-based scheme for KV cache compression can be implemented. This scheme preserves important terms and applies unique techniques for managing unimportant terms. The centroids of clusters created at different clustering levels with variable precision can approximate the key and value tensors corresponding to unimportant terms. Specifically, the centroids of key and value tensors in the clusters closest to the centroids (or the centroids of the clusters to which the key or value tensors belong) can approximate the key and value tensors. The centroids can be used as surrogate key and value tensors for unimportant terms. This approximation reduces the precision loss due to compression. Furthermore, since more significant attention heads have a greater impact on model accuracy than less significant attention heads, precision loss can be reduced by using surrogates generated at finer-grained clustering levels with sub-levels of more significant attention heads. Furthermore, fast (and small) memory can store proxies generated at a coarser granularity (e.g., centroids generated using a smaller number of clusters), while slow (and large) memory can store proxies generated at a finer granularity (e.g., centroids generated using a larger number of clusters). When the impact on accuracy is low, latency can be improved by retrieving proxies generated at a coarser cluster level using a subset of less significant attention heads from faster memory.

[0024] Instead of examining or scanning each cluster level to find the closest surrogate key and value tensors to represent the computed key and value tensors, the cluster level of the generated surrogate key and value tensors is determined (directly) based on the saliency score of the attention head used to compute the key and value tensors. Specifically, the cluster level is selected, chosen, or determined from multiple different cluster levels using the saliency score of the attention head. The saliency score of the attention head can be measured based on the similarity between the input and output of the attention head. In one embodiment, the cosine similarity between the input and output of the attention head can be used to determine the saliency score of the attention head. The input can be a query tensor, and the output can be the output of the attention function. Different ranges of the saliency score of the attention head can be mapped to different cluster levels at various granularities. The saliency score of the attention head can thus be directly mapped to a specific cluster level for surrogate retrieval, and the mapping can be completed in one step to determine the specific cluster level. The surrogate key and value tensors of the attention head can be retrieved in a corresponding memory storing the centroids of the clusters generated according to the specific cluster level. Using saliency scores to determine the clustering level of the generated surrogate key tensors and surrogate value tensors avoids the additional computation / iteration required to find the closest surrogate and allows for fast surrogate retrieval based on whether the accuracy loss will significantly contribute to the final accuracy of the transformer-based neural network.

[0025] The multi-granularity clustering scheme is hardware-aware. First, different memories with varying speeds / latencies and sizes are initially developed to evaluate the resources available in the underlying memory hierarchy architecture. One or more other factors / considerations can be determined, such as maximum sequence length, batch size, maximum number of concurrent users, maximum number of concurrent requests, tolerable accuracy loss, target inference latency, and / or required throughput. A set of cluster levels can be determined based on one or more factors such as available resources and tolerable accuracy loss.

[0026] Multi-granularity clustering-based schemes offer greater KV cache compression, improved LLM service / inference throughput, and reduced inference latency. Hardware-aware schemes enable multi-granularity clustering-based schemes to effectively utilize the underlying memory hierarchy. The high compression ratio of the KV cache, achieved through multi-granularity clustering, leads to increased inference throughput for managed LLM services. Furthermore, multi-granularity clustering-based schemes are compatible with other KV cache management techniques such as KV cache paging and quantization.

[0027] In some experiments, for a ~3% decrease in accuracy, using a multi-granularity clustering-based approach can achieve approximately 15x compression and more than 50% reduction in inference latency.

[0028] In some experiments, for state-of-the-art large language models with fixed cache budgets, multi-granularity clustering-based schemes have achieved significant accuracy gains when evaluated for different use cases such as question and answer, and text summarization.

[0029] In some experiments, when using a multi-granularity clustering-based scheme, inference latency, specifically the average memory access latency, can be significantly reduced by leveraging the advantages of a cache hierarchy architecture. Furthermore, when compared to schemes that implement clustering techniques for searching for optimal clustering (e.g., k-means clustering), the total cost of clustering can be significantly reduced by using a greedy clustering / grouping algorithm.

[0030] The various implementations described herein can be illustrated within the context of specific architectures or implementations. It is envisioned that the teachings of the implementations described herein can be applied to other neural networks or models that have attention mechanisms or attention heads and can employ KV caching schemes to reduce computation.

[0031] Various embodiments of KV caches are described in the context of multilevel memory systems or multilevel cache systems. It is contemplated that the teachings of the embodiments described herein can be applied to other variations or types of multilevel cache systems or hierarchical cache architectures. It is also contemplated that the teachings of the embodiments can be applied to distributed computer systems / environments with multiple memories of different sizes and / or latencies. Furthermore, it is contemplated that the teachings of the embodiments can be applied to independent computing systems with multiple memories of different sizes and / or latencies. Transformer-based neural networks or transformer-based models

[0032] Generative AI models such as LLMs have swept through the computing industry with tremendous force. These models are equipped with massive amounts of parameters and exhibit top-tier performance across various tasks. The current trend of LLM models is moving towards the scale of models with trillions of parameters. According to one assessment, the number of models is increasing tenfold every two years. The current trajectory makes it practically impossible for small and medium-sized players to operate and serve LLMs, and the sheer size of these models (which only require 325GB of memory to load their model weights) renders traditional optimization techniques such as prefetching, data streaming, and caching completely ineffective. Furthermore, LLMs pose a significant challenge to the platform's computational and memory resources (bandwidth and capacity) during inference. In addition, the stringent latency requirements (on the order of 50-100ms) make delivering high throughput while maintaining latency even more challenging.

[0033] Figure 1 An exemplary LLM 100, implemented as a transformer-based neural network according to some embodiments of the present disclosure, is shown. LLM 100 may include one or more components: a tokenizer 104, a transformer module stack 110 (e.g., shown as transformer module 0, transformer module 1, transformer module 2, ..., transformer N), one or more classifiers 112, and an inverse tokenizer 114. Tokenizer 104 is capable of segmenting input data (e.g., cue 102) into tokens. For example, cue 102 may include text, and tokenizer 104 may segment cue 102 into subwords. One or more tokens, denoted as X 106, can be transformed into embeddings 108 that include high-dimensional input features of transformer module stack 110. Transformer module stack 110 is capable of acquiring knowledge about the input data.

[0034] A transformer module in transformer module stack 110 can include two types of layers equipped with learning parameters: attention layers and feedforward (FFN) layers. Figure 2 An exemplary arrangement of the converter module is shown in the figure. Figure 3The diagram illustrates another exemplary arrangement of the transformer module. The attention layer allows the model to weight the importance of lexical terms based on their contextual relevance and capture lexical dependencies. The attention layer implements the attention mechanism of the transformer module, which captures contextual information by focusing on positions within the sequence. The FFN layer provides non-linear transformations for lexical independence.

[0035] One or more classifiers 112 are capable of producing predicted or generated tokens based on the learned representations of the transformer module stack 110. One or more desegmenters 114 can use tokens to generate the generated text 116.

[0036] The LLM 100 can be used as a framework for modeling complex relationships in text, images, audio, video, point clouds, graphs, and more. Using this framework allows for scaling the number of learning parameters to model even more complex relationships.

[0037] LLM 100 is designed to model sequential text in an autoregressive manner. Each subsequent word, denoted as Y 182, is determined by the context of the preceding word. During training, the transformer architecture's task is to learn to predict the next word Y 182 from text segments with known subsequent words. Leveraging the abundance of text data available on the Internet, the transformer size can be scaled dramatically to hundreds of billions of parameters. LLM 100 can be referred to as an autoregressive transformer, a causal transformer, a pure decoder transformer, and a decoder transformer. The subsequent alignment stage allows LLM 100 to undergo context transformation and be converted to a human-like manner. A regular LLM involving LLM 100 can be referred to as a generative pre-trained transformer (GPT). Aligned LLMs can be referred to as instruction-regulated LLMs, instruction-following LLMs, and supervised fine-tuning LLMs.

[0038] Autoregressive modeling involves sequential prediction during its development; therefore, LLM-based applications generally involve text generation by outputting word by word. The autoregressive nature of the model means that the entire model structure is used for prediction of each word. Due to the massive number of model parameters (currently reaching billions), sequence inference is computationally intensive, characterized by an initial computationally intensive first prediction, followed by subsequent word-by-word predictions limited by memory bandwidth. The computational complexity of the attention layer is quadratic with the sequence length. This complexity severely restricts performance, especially for longer sequences.

[0039] Figure 2A series converter module 200 according to some embodiments of the present disclosure is shown. The series converter module 200 includes an attention layer 204 and an FFN layer 206. First, the input X 202 is processed by the attention layer 204, and the output of the attention layer 204 is passed to the FFN layer 206. The FFN layer 206 can generate an output X' 208. In some cases, the series converter module 200 may include a skip connection that passes the input X 202 to be added to the output X' 210. The series converter module 200 can be implemented as follows: Figure 1 A converter module in the converter module stack 110.

[0040] Figure 3 A parallel converter module 300 according to some embodiments of the present disclosure is shown. The parallel converter module 300 includes an attention layer 304 and an FFN layer 306. The input X 302 is processed by the attention layer 304 and by the FFN layer 306 (in parallel). The outputs of the attention layer 304 and FFN layer 306 are combined at an adder 308. The adder 308 can generate a sum of its inputs, for example, the sum of the outputs of the attention layer 304 and FFN layer 306. The adder 308 can also generate a weighted sum of its inputs, for example, a weighted sum of the outputs of the attention layer 304 and FFN layer 306. The adder 308 can generate an output X'310. In some cases, the parallel converter module 300 may include a skip connection that passes the input X 302 to the output X'310. The parallel converter module 300 can be implemented as... Figure 1 A converter module in the converter module stack 110.

[0041] like Figure 1 The transformer-based neural network model shown here relies on an encoder-decoder stack with the same layers. Figures 2 to 3 As shown, each layer can have two key components: self-attention and a feedforward network. Self-attention allows the model to analyze the entire sequence at once; however, a single mechanism may miss subtleties. Multi-head attention addresses this problem by creating multiple independent "heads" that focus on different aspects of the relationships between words. Figure 4 The diagram illustrates a multi-head attention mechanism. The outputs from these heads are combined to gain a richer understanding. Feedforward networks supplement self-attention by introducing non-linearity, enabling the model to learn complex patterns. The number of layers stacked in the encoder and decoder (depth) and the number of heads within each layer (width) are hyperparameters. More layers and heads improve the model's ability to capture long-range dependencies, but increase complexity. The heads and layers of the individual transformer modules work together to allow the LLM to provide nuanced control over textual data, resulting in superior performance in natural language processing tasks.

[0042] Figure 4 illustrates an attention layer 400 of a converter module according to some embodiments of the present disclosure. The attention layer 400 may be included as... Figure 1 This is part of a transformer module in the transformer module stack 110. As an embodiment, attention layer 400 illustrates a multi-head attention layer with multiple attention heads. Input X 402 is transformed into a query (Q), a key (K), and a value (V). Attention layer 400 includes using a query weight matrix W... Q The query is subjected to parallel linear projection 404. The attention layer 400 includes the use of the key weight matrix W. K The parallel linear projection of the keys 406. The attention layer 400 includes a weight matrix W. V The values ​​are then projected into a parallel linear projection 408. The result of the linear projection is provided to a parallel attention head 410. The attention head 410 can apply an attention function using the outputs from a linear projection 404, a linear projection 406, and a linear projection 408. The attention function can be defined as follows:

[0043] In Equation 1, Q represents the output from a linear projection 404. In Equation 1, K represents the output from a linear projection 406. In Equation 1, V represents the output from a linear projection 408. k Indicates the expansion factor. Attention head 410 can be calculated based on the query and key. This is used to generate the matrix of raw attention scores. Attention head 410 can be calculated. This generates the attention weight matrix, which is then a normalized matrix of the original attention scores. The attention head 410 can be calculated... This generates the final output, in which the attention weights are weighted by values ​​to form the final attention representation.

[0044] The outputs of the parallel attention head 410 can be concatenated and used with the output matrix W. O It is passed to linear projection 412. The output of linear projection 412 is the output X'414 of attention layer 400.

[0045] The linear projection used in attention layer 400 may include multiplying the input of the linear projection with the learned weight matrix. In some cases, the matrix multiplication is followed by an optional nonlinearity, such as an activation function.

[0046] like Figure 1The attention mechanism discussed is a major performance bottleneck for long sequences in models based on autoregressive transformers. KV caching can be configured to store key and value tensors previously computed from the attention mechanism and reuse these cached tensors to generate the current term, thereby avoiding the cumbersome recomputation of key and value tensors for previous terms. KV caching becomes a factual optimization of the inference process to accelerate LLM throughput generation, allowing attention operations to scale linearly, rather than quadratically, with the total sequence length. Figure 5 and Figure 6 The computation in the attention layer with and without KV cache was compared. Understanding KV Cache

[0047] Figure 5 Computation in a KV-free self-attention layer according to some embodiments of this disclosure is illustrated. The self-attention layer may be part of a multi-head self-attention layer. In some embodiments, the self-attention layer is located in the decoder of the transformer. In some embodiments, the self-attention layer may be located in the transformer module, such as in... Figure 1 In the transformer module shown, computations in the self-attention layer may include multiplication of the query matrix 510 with the key matrix 520 (having one or more key tensors), which generates an attention weight matrix 530. Computations in the self-attention layer also include multiplication of the attention weight matrix 530 with the value matrix 540 (having one or more value tensors), which generates an output matrix 550 encoding new lexical units such as lexical unit 5. In some cases, the output matrix 550 may include a context-aware attention representation weighted by the value matrix 540. The output matrix 550 may be generated by the attention layer using the query matrix 510, key matrix 520, and value matrix 540, which are capable of generating one or more new lexical units, according to Equation 1. In other embodiments, computations in the self-attention layer may include other computations such as those with scaling functions, SoftMax functions, etc. For simplicity and illustration purposes, not shown... Figure 5 These calculations are shown in the figure.

[0048] Each of the query matrix 510, key matrix 520, and value matrix 540 may include a tensor (e.g., a vector) for each term in the input sequence. For illustrative and concise purposes, the input sequence has four terms: terms 1-4. Query matrix 510 may include four query tensors generated based on the four input terms: query tensors 1-4. Key matrix 520 may include four key tensors: key tensors 1-4. Value matrix 540 may include four value tensors: value tensors 1-4. Figure 5In this implementation, since the decoder does not implement KV caching, it is necessary to perform calculations for the key tensors in the key matrix 520 and all value tensors in the value matrix 540. Some calculations have already been performed in the previous inference phase, such as the calculations for generating key tensors 1-3 and value tensors 1-3. This duplication of calculations may result in a waste of computational resources such as computing power and time.

[0049] Figure 6 Computation in a self-attention layer with KV buffering according to some embodiments of this disclosure is illustrated. Unlike Figure 5 Implementation method, Figure 6 The decoder in the code implements a key-value (KV) cache. Using the KV cache, key and value tensors computed in previous inference stages (e.g., key and value tensors corresponding to terms 1-3) are cached in the KV cache and can be reused in the current inference stage. The KV cache stores previously computed key and value tensors computed for one or more terms in the attention mechanism and reuses these previously computed key and value tensors to generate the next attention output or term. In implementations where the neural network is executed on distributed GPU worker nodes, the KV cache can be allocated in GPU memory, and its contents can be loaded from the central processing unit (CPU) memory. In implementations where the neural network is executed on a processor, the KV cache can be allocated in one or more local memories of the processor. When using a KV cache, execution time scales more gradually with increasing sequence length. For example, intermediate KV tensors corresponding to previous terms can be stored in the KV cache.

[0050] exist Figure 6 In the current inference phase shown, cached key and value tensors can be retrieved from the KV cache. Data that can be retrieved from the KV cache is... Figure 6The dotted line pattern is highlighted. Key tensors 1-3 can be retrieved from the KV cache. Value tensors 1-3 can also be retrieved from the KV cache. In the current inference phase, query matrix 510 can be multiplied by the concatenation of cached key tensors 1-3 with key tensor 4, followed by performing a SoftMax on the entire original attention score. The attention weights generated by performing a SoftMax on the original attention score can be further multiplied by the concatenation of value tensor 4 and cached value tensors 1-3 to generate new results. After inference is complete, key tensor 4 is added to the KV cache. In some cases, key tensors 1-3 can be updated in the KV cache. Additionally, value tensor 4 is added to the KV cache. In some cases, value tensors 1-3 can be updated in the KV cache. This process is repeated for each term. The KV cache reduces the computational cost in the self-attention layer. The computational cost is significantly reduced when cached key and value tensors can be reused to generate the next term. Therefore, computational resources can be saved. Key-value caching can improve the performance and efficiency of transformer models.

[0051] like Figure 6 As shown, when using a KV cache, previously computed key-value tensors are stored in memory (e.g., the KV cache) to avoid redundant key-value projection computations in the attention mechanism. The total memory footprint for a KV cache instance can be easily calculated using Equation 2: Size = 2 × precision × n layers ×d model ×L sequence ×B (Equation 2)

[0052] precision is the number of bytes for each value that has been stored (e.g., B for FP32), n layers d represents the number of layers in the model. model L represents the dimension of the embedding. sequence B is the length of the context in the lexicon, B is the batch size, and a factor of 2 is applied because two matrices are needed for the keys (K) and values ​​(V).

[0053] As shown in Equation 2, the KV cache size scales linearly with the (maximum) sequence length and batch size in the input context. In practice, the KV cache size can be enormous. For example, a transformer-based model with 175 billion parameters can consume approximately 325 GB of memory to store the parameters. Meanwhile, with a batch size of 128 and a sequence length of 8K, the KV cache can have a memory size of approximately 4608 GB, which is several orders of magnitude (12X) larger than the model weights themselves. Because the total sequence length cannot be known in advance, the KV cache requirement is therefore unknown, making LLM memory management particularly challenging. Typically, the maximum sequence length (usually 4K and growing rapidly) is used for memory allocation to host the KV cache, resulting in severe memory fragmentation and very low batch sizes, and therefore, a small number of concurrent users are feasible for LLM services.

[0054] The size of the key-value (KV) cache is becoming increasingly prominent and is a key factor making LLM model development very expensive. Reducing the memory footprint of the KV cache in an LLM without sacrificing accuracy is challenging. As extending sequence length becomes a significant requirement for many enterprises, limiting the context sequence becomes unpredictable. The only design knob available for LLM development scaling according to Equation 2 is the batch size (B). In practice, reducing the batch size reduces model throughput, thus significantly decreasing the total number of requests per second that the model can serve. Related work in KV cache compression

[0055] As discussed above, the development of LLMs for generating long-context lexics is challenged by high memory requirements. This is primarily due to the need to store all previous lexics in the attention module, resulting in a large memory footprint for the KV cache. Some methods have achieved a smaller memory footprint for the KV cache using schemes such as lexical pruning and approximation.

[0056] Lexical pruning schemes prioritize retaining a subset of lexical units considered crucial to the attention mechanism. The salience of these lexical units is typically determined by factors such as attention weights or their proximity to the current lexical unit. Some schemes focus on retaining only a selected subset of lexical units that contribute most to the attention mechanism. Lexical unit selection can be based on attention weights (which represent the lexical unit's contribution) or on the lexical unit's proximity to the currently processed lexical unit. Some methods leverage the observation that a small subset of lexical units contributes the majority of the value when calculating attention scores / weights. Some methods are based on the observation that lexical importance decreases exponentially with increasing distance from the current lexical unit. Some lexical elimination schemes address this issue by excluding lexical units considered unimportant to the attention algorithm based on factors such as attention weights or distance from the current lexical unit. However, these schemes often suffer from a loss of accuracy, or even a significant reduction in the number of lexical units.

[0057] Techniques for approximating key and value tensors aim to preserve all terms through various strategies, including matrix approximation and mixed-precision quantization. Some schemes preserve key-value caches corresponding to important terms and strategically manage key-value caches corresponding to less important terms. Some methods can incorporate techniques such as adding control noise, approximating larger matrices with smaller matrices, and reducing the precision of the data representation (mixed-precision quantization). One method introduces Gumbel noise to preserve weakened versions of unimportant terms. Another method employs mixed-precision quantization to reduce the memory footprint of unimportant terms. A third method employs and utilizes low-rank matrix approximation. Multilevel memory system

[0058] Figure 7 A multi-level memory system according to some embodiments of the present disclosure is illustrated. The computing core 702 may be communicatively coupled to multiple (non-transitory computer-readable) memories, such as memory 704, memory 706, memory 708, memory 710, etc. The computing core 702 may be an embodiment of processing device 1302, and the memory may be... Figure 13An embodiment of memory 1304. Computational kernel 702 may include a processor or computational logic that processes requests to generate one or more terms based on one or more input terms. Computational kernel 702 may execute instructions to perform one or more operations or computations of a transformer-based neural network, such as computing an attention function of an attention mechanism for a transformer-based neural network. Memory may store data used by computational kernel 702 to perform one or more operations. When one or more operations are completed, memory may store data computed by computational kernel 702. The data may include a key-value (KV) cache. The KV cache may include one or more cached key tensors and one or more cached value tensors corresponding to one or more terms. Memory may store the KV cache and provide cached key tensors and cached value tensors to computational kernel 702 to accelerate the processing of the attention mechanism by avoiding redundant computations and reusing previously computed key tensors and value tensors.

[0059] The memory can be used to store data (in some cases, instructions) used by the computing core 702. The memory can be arranged in a hierarchical architecture or a multi-level memory system. Multi-level memory systems are designed to balance speed / latency, cost, and capacity. The hierarchical architecture can range from fastest / minimum / most expensive to slowest / maximum / cheapest. In the illustrated embodiment, memories 704, 706, 708, and 710 are arranged from fastest / minimum / most expensive to slowest / maximum / cheapest. The fastest / minimum / most expensive memory may include static random access memory (SRAM). The slowest / maximum / cheapest memory may include dynamic random access memory (DRAM). For example, memories 704, 706, and 708 may be SRAM, and memory 710 may include DRAM. In some scenarios, memories 704, 706, and 708 may be L1 caches, L2 caches, and L3 caches that operate to bridge the gap between the computing core 702 and memory 710, which operates as main memory. The size / capacity of the memory and the latency of the memory can gradually increase from memory 704 forward to memory 710. Hardware-aware key-value caching compression

[0060] Figure 8A multi-level memory system with KV cache compression according to some embodiments of the present disclosure is illustrated. A cache controller 802 can be implemented on a computing core 702 or on a separate computing processor communicatively coupled to the memory, such as memory 704, memory 706, memory 708, and memory 710. The cache controller 802 can control what data is retained / stored in the various memories in a hardware-aware manner. The cache controller 802 can control what data is evicted from the various memories. The cache controller 802 can control memory utilization. The cache controller 802 can implement a KV cache on the memory so that cached key tensors and cached value tensors can be provided to the computing core 702. The cache controller 802 can implement one or more memory management techniques, such as KV cache compression, to efficiently utilize the memory.

[0061] The cache controller 802 may include discovery 804 and cluster-level allocation 806. Discovery 804 provides information to cluster-level allocation 806 to determine the appropriate cluster level for different memories in the hierarchical architecture. Discovery 804 and cluster-level allocation 806 enable better development, such as... Figure 7 The diagram shows the memory hierarchy architecture in the underlying hardware (different memory tiers with different sizes and access latencies). The cluster level allocation 806 allows adjustment of the design knob by setting the cluster level of memory to balance precision and latency. Here, the cluster level can define the granularity of clustering and / or the number of clusters to create when clustering is performed.

[0062] Discovery 804 has the ability to discover different memory types. Discovery 804 can determine various speeds / latencies, and / or sizes, as well as the amount of memory available for storing KV caches. Discovery 804 can also determine other factors / considerations, such as maximum sequence length, batch size, maximum number of concurrent users on the system, maximum number of concurrent requests, tolerable precision loss, target inference latency, and / or required throughput. Discovery 804 can discover one or more factors / considerations that may affect how best to utilize memory.

[0063] Cluster-level allocation 806 can determine the cache budget for retaining important terms based on one or more factors / considerations, such as those determined by discovery 804. Cluster-level allocation 806 can also determine multiple cache budgets for storing surrogate key tensors and surrogate value tensors for unimportant terms, based on one or more factors / considerations, such as those determined by discovery 804. A higher cache budget can be set for memory storing one or more centroids or representatives of clusters generated at a finer-grained clustering level. A lower cache budget will be set for memory storing one or more centroids or representatives of clusters generated at a coarser-grained clustering level. The cache budget can be determined by the available resources of the underlying hardware. The cache budget used to store surrogate key tensors and surrogate value tensors is directly related to the clustering level, which determines the number of clusters to be created and the number of centroids / surrogates to be stored in a given memory during the clustering process.

[0064] Cluster level allocation 806 determines a set of cluster levels, such as C cluster levels, by optimizing performance while satisfying one or more factors / considerations. In some cases, the cluster level directly depends on the cache budget allocated for a given memory. C can be an adjustable parameter. C can be determined based on the depth of the memory hierarchy architecture in the underlying hardware (discovered by discovery 804). C cluster levels can include: cluster level 1, cluster level 2, cluster level 3, ..., cluster level C, etc. This set of cluster levels can include the cluster level for each memory used to store KV cache data. This set of cluster levels can include different cluster levels from the coarsest to the finest granularity. The granularity of clustering can become finer from level 1 to level C. Finer granularity means generating more clusters. Because coarser granularity means generating fewer clusters, and finer granularity means generating more clusters, the granularity of the cluster level has a direct impact on how much capacity is needed on memory. Fewer clusters mean fewer centroids or representatives used as proxies retained / stored in memory and mean higher KV compression. More clusters mean more centroids or representatives used as surrogates are retained / stored in memory and imply lower key-value compression. Because coarser granularity means the surrogate is more likely to be used as a better approximation, and finer granularity means the surrogate is less likely to be used as a better approximation, the granularity at the cluster level has a direct impact on the accuracy of the surrogate.

[0065] Brief Reference Figure 9 , Figure 9 This shows different clustering levels within a set of clustering levels. Figure 8The clustering level allocation 806 can allocate clustering levels to memory, or allocate a set of clustering levels to memory arrangements ranging from fastest / smallest to slowest / largest. The described embodiment has three clustering levels: clustering level 1, clustering level 2, and clustering level 3. Clustering level 1 has the coarsest granularity (only one cluster is shown), clustering level 2 has a finer granularity (two clusters are shown), and clustering level 3 has the finest granularity (four clusters are shown). The underlying hardware in this embodiment may include three memories in a hierarchical architecture shown as memory 1, memory 2, and memory 3. Memory 1 may be the fastest / smallest memory. Memory 2 may be a slower / larger memory. Memory 3 may be the slowest / largest memory. Return to Reference Figure 8 , Figure 8 The clustering level assignment 806 can assign clustering level 1 to memory 1, clustering level 2 to memory 2, and clustering level 3 to memory 3. Even though each memory corresponds to a different clustering level, each memory stores surrogate or representative tokens for all terms. Any memory can be used to retrieve surrogate or representative tokens. Memory can correspond to unimportant terms. Because the initially computed key tensors and value tensors are not retained in memory, and only surrogate or representative tokens are retained, compression is performed in these memories. Finer-grained clustering levels provide a better approximation, but at the cost of lower KV compression.

[0066] Return to reference Figure 8 The clustering level allocation 806 can determine an appropriate clustering level based on memory capacity, allowing surrogate key tensors and surrogate value tensors to be designed to reside in their respective memories. This clustering level allocation is used to compress the KV cache at different compression levels. Key tensors and value tensors will be clustered at different granularities with varying levels of precision.

[0067] Cluster-level allocation 806 can allocate one or more memories (e.g., smaller / faster memories) in the hierarchical architecture to store surrogate key tensors and surrogate value tensors corresponding to unimportant terms.

[0068] Cluster-level allocation 806 can allocate a memory (e.g., maximum / slowest memory) in the hierarchical architecture to store key tensors and value tensors corresponding to important terms. The memory can correspond to important terms. The memory can have "non-clustering" as the clustering level.

[0069] The surrogate calculator 808 can determine surrogates or representatives stored in memory at different clustering levels. The surrogate calculator 808 can evaluate the key and value tensors of all terms and perform clustering at different clustering levels. The technical task of the surrogate calculator 808 is to determine or select a surrogate or representative that provides an approximation of the key and value tensors corresponding to a set of terms. Using surrogates or representatives means that the KV cache can avoid having to store the entire set of key and value tensors. The surrogate key and surrogate value tensors can thus be used as approximations of the initially computed key and value tensors. The surrogate calculator 808 can determine one or more surrogate key tensors and one or more surrogate value tensors based on clusters generated at a specific clustering level and store the one or more surrogate key tensors in memory for which that specific clustering level is assigned.

[0070] In some implementations, the proxy calculator 808 can group key tensors and value tensors into one or more clusters based on a specific clustering level, such that similar key tensors and value tensors corresponding to a set of terms are grouped together. The proxy calculator 808 can perform clustering or grouping of key tensors and value tensors to generate clusters corresponding to different groups of terms. Suitable clustering algorithms, i.e., suitable for high-dimensional vectors, can be used to cluster similar tensors into spatial groups. One embodiment is the K-means clustering algorithm. Another embodiment is density-based noise-applied spatial clustering (DBSCAN). In some cases, greedy clustering or grouping clustering can be used by the proxy calculator 808 to generate reasonable clustering results based on one or more heuristics. Greedy clustering or grouping algorithms may involve (randomly) selecting centroids as the centers of clusters and iteratively evaluating all points using the point farthest from the existing centroids to add another centroid as the center of another cluster, until all the required number of centroids are set. Once clusters are generated, the surrogate calculator 808 can use clusters corresponding to a set of terms to select or compute surrogates. The centroids of clusters corresponding to a set of terms can be computed and used as surrogates or representatives that approximate the key and value tensors corresponding to that set of terms. The key and value tensors closest to the centroids of the clusters corresponding to a set of terms can be selected and used as surrogates or representatives that approximate the key and value tensors corresponding to that set of terms. Key and value tensors in clusters corresponding to a set of terms can be randomly selected and used as surrogates or representatives that approximate the key and value tensors corresponding to that set of terms.

[0071] In some implementations, the proxy calculator 808 can cluster one or more key tensors and one or more value tensors according to different clustering levels. The proxy calculator 808 can determine proxy key tensors and proxy value tensors using one or more clusters at a given clustering level. For example, the proxy calculator 808 can determine a proxy based on one or more centroids of one or more clusters at a given clustering level. A proxy can be determined for each clustering level. The proxy calculator 808 can store the proxy key tensors and proxy value tensors in memory corresponding to a given clustering level, for example, memory allocated by clustering level allocation 806 to store proxies generated at a given clustering level. The proxy key tensors and proxy value tensors can be provided to the processor for reuse when a processor performing one or more operations of a neural network generates one or more subsequent terms.

[0072] When new key tensors and new value tensors for new terms are generated (e.g., by computational kernel 702 via an autoregressive process) and added to the KV cache, agent update 830 can be implemented as updating agents or representatives stored in memory. Agent update 830 can perform re-clustering or regrouping of terms using a suitable algorithm to form new clusters of terms corresponding to different groups. In some cases, to avoid re-clustering or regrouping (which may be computationally expensive), agent update 830 can compare the distances of the new key tensors and new value tensors to existing agents or representatives to determine whether the new key tensors and new value tensors belong to existing clusters. The distances can be compared with a threshold. If the distance to an existing agent / representative is below the threshold, the new key tensors and new value tensors may belong to the existing cluster represented by the existing agent / representative. If the distance to an existing agent / representative is above the threshold, the new key tensors and new value tensors may not belong to the existing cluster represented by the existing agent / representative. If the new key tensor and the new value tensor do not belong to an existing cluster (or any existing cluster), the new key tensor and the new value tensor can form their own new cluster and can be used as surrogates / representatives for that new cluster. If the new key tensor and the new value tensor belong to an existing cluster (or at least one existing cluster), the existing surrogate or representative representing the existing cluster may become the surrogate or representative of the new key tensor and the new value tensor. If appropriate, the surrogate or representative of the existing cluster can be updated based on the new key tensor and the new value tensor (e.g., the centroid of the cluster can be moved toward the new key tensor and the new value tensor). In some implementations, the distance of the new key tensor and the new value tensor to the existing surrogate or representative can be compared with a threshold to determine whether the new key tensor and the new value tensor belong to an existing cluster. If the distance does not exceed the threshold (e.g., the distance is less than the threshold), the existing surrogate or representative (or a derivative thereof) is used to represent the new key tensor and the new value tensor. If all distances exceed the threshold (e.g., all distances are greater than the threshold), then the new key tensor and the new value tensor can become the new cluster and be used as proxies or representatives of the new cluster.

[0073] Lexical importance checker 810 can determine / evaluate whether a lexical is important. A lexical may be part of a request for a neural network, such as a transformer-based neural network. Lexical importance checker 810 can determine that a lexical is important based on its importance score or importance. Lexical importance checker 810 can determine that a lexical (otherwise) is unimportant based on its importance score or importance. Lexical importance checker 810 can determine the importance score or importance of a lexical based on attention weights or other metrics associated with the lexical.

[0074] like Figure 5 and 6As shown, the attention mechanism operation performed by computation kernel 702 determines the attention weight matrix based on query tensors and key tensors corresponding to different terms. The attention weight matrix can be arranged with rows corresponding to different query terms and columns corresponding to different key terms (or columns corresponding to different query terms and rows corresponding to different key terms). Referring to Equation 1, the attention weight matrix can include computation... Or the result of its derivatives. The lexical importance checker 810 can use the calculated attention weight matrix to determine the attention weight corresponding to a specific lexical. The attention weight can provide an indication of the contribution / importance of a specific lexical to the attention mechanism, or an indication of how much attention the attention mechanism has given to a specific lexical. In some cases, the contribution / importance (or the attention weight corresponding to a specific lexical) can be inversely biased against the distance from the specific lexical to the currently processed lexical. The lexical importance checker 810 can determine whether a lexical is important or unimportant based on its attention weight. In some implementations, determining whether a lexical is important or unimportant includes comparing the attention weight corresponding to the lexical with a threshold. The threshold can be a hyperparameter set based on the probability distribution of the attention weights. The threshold can be set to classify a specific percentage of lexicals as important and a specific percentage of lexicals as unimportant. A lexical can be determined as important in response to an attention weight exceeding or surpassing the threshold. A lexical can be determined as unimportant in response to an attention weight below the threshold.

[0075] The response lexical importance checker 810 determines that a lexical is important, and the KV tensor retention 812 can retain one or more key tensors and one or more value tensors corresponding to the important lexical. The one or more key tensors and one or more value tensors can be computed by the attention head of the neural network. The KV tensor retention 812 can store the key tensors and value tensors corresponding to the important lexical in memory to retain the important lexical. The memory can be configured by cluster-level allocation 806 to correspond to the memory of the important lexical. When the computation processor generates one or more additional lexicals, the key tensors and value tensors corresponding to the important lexical can be provided from memory to the computation processor for reuse.

[0076] The response lexical importance checker 810 determines that a lexical is unimportant by pruning one or more key tensors and one or more value tensors corresponding to the unimportant lexical by using one or more surrogate key tensors and one or more surrogate value tensors as proxies or representatives, rather than the originally computed key tensors and value tensors. Key tensors and value tensors corresponding to unimportant lexical may be evicted or discarded.

[0077] After determining that the lexicon is unimportant, the new operation is to determine the cluster level, which agent is used for the unimportant lexicon, or which memory in the memory to obtain the agent from.

[0078] In one implementation, the technical task is to select surrogate key and surrogate value tensors from shallower cache levels (faster memory). Subsequent (or slower / deeper) cache levels (slower memory) should only be accessed if the surrogates are not sufficiently close. To find the surrogate key and surrogate value tensors for a given term, the coarsest-grained clustering level can be accessed first, and subsequent finer-grained clustering levels can be accessed continuously until the surrogates (e.g., selected centroids within a clustering level) are sufficiently close. This implementation can optimize or aims to improve average access latency without affecting model accuracy.

[0079] Scanning cluster levels to find the most accurate agent to use can have a significant impact on inference latency. Different implementations avoid or eliminate scanning cluster levels or shallower caches to reach specific cluster levels or memories. Instead, the search for or determination of cluster levels is optimized by mapping (all) terms belonging to an attention head within a layer to a cluster level based on the attention head's saliency score. The attention head of a neural network can compute or process key and value tensors corresponding to unimportant terms. A head saliency score calculator 814 can calculate the saliency score of the attention head. The saliency score can be calculated by the head saliency score calculator 814 based on the similarity between the attention head's input and output, such as the cosine similarity between the attention head's input and output. Instead of performing a stepwise search of cluster levels, the attention head's saliency score determines which memory to store the agent, which agent to use, or which cluster level the agent should use.

[0080] In some implementations, the head saliency score calculator 814 can calculate the saliency score by calculating the cosine similarity between the input and output of the attention head. The cosine similarity can be calculated using the following equation: cosine similarity=(I·O) / (||I||×||O||) (Equation 3)

[0081] I is the input (vector) of the attention head. O is the output (vector) of the attention head. I·O is the dot product of the input and output of the attention head, and ||A|| and ||B|| are the magnitudes (lengths) of the vectors. The input of the attention head can be replaced with I in Equation 3 and the output of the attention head can be replaced with O in Equation 3. In some cases, the head saliency score calculator 814 can determine one or more other measures of similarity, such as Euclidean distance, Manhattan distance (or L1 distance), Pearson correlation, Jaccard similarity, etc.

[0082] In some cases, I corresponds to Q in Equation 1, and O corresponds to attenton(Q, K, V) in Equation 1. Cosine similarity can compare two vectors of the same size, and Q and Attenton(Q, K, V) can be vectors of the same size. In other words, cosine similarity measures the input and output of the attention head.

[0083] The saliency score clustering level mapper 816 can select or determine (or use for selection or determination) a clustering level from multiple different clustering levels of the attention head based on the saliency score determined by the head saliency score calculator 814. Different clustering levels can correspond to different ranges or subranges of saliency scores. Low similarity can indicate that the output of the attention head is significantly different from the input of the attention head. Low similarity can indicate that the computation performed by the attention head is significant or can make a significant contribution to the accuracy of the neural network. For this reason, the saliency score clustering level mapper 816 can map attention heads with low similarity scores to finer-grained clustering levels to prevent meaningful accuracy loss. The saliency score clustering level mapper 816 can map attention heads with high similarity scores to coarser-grained clustering levels.

[0084] A given cluster level can correspond to a subrange of significance scores. Different cluster levels can correspond to different subranges of significance scores. The significance score cluster level mapper 816 can determine the cluster level by determining that the significance score falls within a subrange of significance scores. In one embodiment, the mapping from the range of significance scores to different cluster levels is as follows:

[0085] In one embodiment, an attention head with low similarity or high significance scores can be mapped to cluster level C. An attention head with moderate similarity or moderate significance scores can be mapped to cluster level C-1. An attention head with high similarity or low significance scores can be mapped to cluster level 1. Cluster level 1 can be the coarsest cluster level. Cluster level 2 can be a finer-grained cluster level. Cluster level C can be the finest-grained cluster level.

[0086] Advantageously, the mapping (e.g., how to define subranges) can be an adjustable design knob that can be adjusted to balance latency and accuracy.

[0087] Using a cluster-level proxy 818, surrogates can serve or provide surrogate key tensors and surrogate value tensors corresponding to unimportant terms from a memory corresponding to the cluster level determined by the cluster-level mapper 816. The surrogate key tensors and surrogate value tensors can be reused by a computational processor that performs the mapping to the attention head at the cluster level to generate the next term. By serving or providing proxies from the memory of the specific cluster level mapped to the saliency score of a given attention head, the precision loss from term pruning is reduced by using proxies generated at a coarser granularity with less significant attention heads and by using proxies generated at a finer granularity with more significant attention heads. Methods for KV cache compression

[0088] Figure 10 This is a flowchart illustrating KV cache retention and approximation using an agent according to some embodiments of this disclosure. Method 1000 can be implemented to provide a scheme based on multi-granularity clustering. It can use methods such as... Figure 13 The computing device 1300 in the computing device executes the method 1000.

[0089] In method 1002, input cues from transformer-based neural networks can be processed. The key and value tensors generated for the words in the input cues can be clustered at different clustering levels. Method 1000 can proceed to the current word processing.

[0090] In step 1004, determine whether the current term is important. The importance of a term can be determined based on its significance. If the current term is important, it is not pruned. If the current term is unimportant, it is pruned; for example, if the key and value tensors computed for the current term are not stored in the KV cache, then surrogate key and value tensors are stored in the KV cache instead. The importance of the current term can be determined based on its attention weight. If the attention weight is greater than or equal to a threshold, the term is important. If the attention weight is less than the threshold, the term is unimportant or insignificant. If the term is important, method 1000 can proceed to step 1006. If the term is unimportant, method 1000 can proceed to step 1008.

[0091] In 1006, important terms are preserved. The key and value tensors corresponding to important terms are preserved or stored in memory designed to store / cache key and value tensors of important terms.

[0092] In step 1008, unimportant terms are pruned, and surrogates are used for unimportant terms. It is possible to determine the saliency or saliency score of the attention heads that generate the key and value tensors corresponding to unimportant terms.

[0093] In 1010, the cluster level of the attention head is determined based on the saliency score. The cluster level of the attention head can be determined by using the mapping between subranges of saliency scores and different cluster levels.

[0094] At 1012, the centroids or representatives at the cluster level can be used as surrogate key tensors and / or surrogate value tensors for unimportant terms. When generating the next term, the surrogate key tensors and / or surrogate value tensors can be used in the computation performed by the attention head.

[0095] For less important terms, 1008, 1010, and 1012 represent the process used to determine which cluster level to retrieve the agent for a given attention head. Transformer-based neural networks can have multiple attention layers, and within each attention layer, the network can include one or more attention heads. The saliency scores of individual attention heads within a single layer or across layers can differ. This means that for attention heads with different saliency scores, agents generated at various cluster levels can be used to balance latency and accuracy.

[0096] Figure 11 This is a flowchart illustrating the determination of the salience of an attention head according to some embodiments of this disclosure. Method 1100 illustrates the operation of 1008 with respect to method 1000. Using methods such as Figure 13 The computing device 1300 in the middle is capable of executing method 1100.

[0097] In 1102, the cosine similarity between the input tensor and the output tensor of the attention head is determined. The input tensor of the attention head may include the query tensor. The output tensor of the attention head may include an output matrix of attention weights weighted by the value tensor (e.g., from the result of calculating Equation 1).

[0098] In 1104, the saliency or saliency score of an attention head can be determined based on cosine similarity. In some cases, the saliency score is the cosine similarity. In other cases, the saliency score is the cosine similarity normalized across all attention heads.

[0099] Figure 12 This is a flowchart illustrating a method for an approximate KV cache based on multi-granularity clustering, according to some embodiments of the present disclosure, having a KV cache associated with unimportant terms. Using methods such as... Figure 13 The computing device 1300 in the middle is capable of executing method 1200.

[0100] In 1202, it can be determined that the lexical unit is unimportant. Pruned lexical units can be determined. Lexical units are part of a request for neural networks such as transformer-based neural networks or neural networks with attention mechanisms.

[0101] In 1204, calculate the saliency score of the attention head of the neural network.

[0102] In 1206, a clustering level is selected from multiple different clustering levels of the attention head based on the significance score.

[0103] In 1208, the surrogate key tensors and surrogate value tensors generated at the cluster level are stored in one of one or more memories. The one or more memories can store data at different cluster levels. The data may include data for KV caching. Multiple memories corresponding to different cluster levels may exist. The surrogate key tensors and surrogate value tensors can represent the key tensors and value tensors computed by the attention heads of the lexical units. For example, the surrogate key tensors and surrogate value tensors can approximate the key tensors and value tensors computed by the attention heads of the lexical units.

[0104] In section 1210, surrogate key and surrogate value tensors are provided to the computational logic that performs one or more operations on the attention head. The computational processor can use the surrogate key and surrogate value tensors to perform the attention head operations. The surrogate key and surrogate value tensors facilitate the reuse of key and value tensors by providing approximations to them and avoid redundant computations in the attention head when generating the next lexical. Exemplary computing device

[0105] Figure 13 This is a block diagram of an apparatus or system, such as an exemplary computing device 1300, according to some embodiments of the present disclosure. One or more computing devices 1300 may be used to implement the functions shown in the figures and described herein. Figure 13 The various components shown can be included in computing device 1300; however, any one or more of these components may be omitted or duplicated depending on the application. In some embodiments, some or all of the components included in computing device 1300 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated as a single system-on-a-chip (SoC) wafer. Furthermore, in various embodiments, computing device 1300 may not include... Figure 13 The computing device 1300 may include one or more components shown, and may include interface circuitry for coupling to one or more components. For example, the computing device 1300 may not include a display device 1306, but may include display device interface circuitry (e.g., connector and driver circuitry) to which the display device 1306 may be coupled. In another set of embodiments, the computing device 1300 may not include an audio input device 1318 or an audio output device 1308, but may include audio input or output device interface circuitry (e.g., connector and support circuitry) to which the audio input device 1318 or the audio output device 1308 may be coupled.

[0106] Computing device 1300 may include processing device 1302 (e.g., one or more processing devices, one or more processing devices of the same type, or one or more processing devices of different types). Processing device 1302 may include electronic circuitry that processes electronic data from data storage elements (e.g., registers, memories, resistors, capacitors, qubit units) to convert the electronic data into other electronic data that can be stored in registers and / or memories. Embodiments of processing device 1302 may include CPUs, GPUs, quantum processors, machine learning processors, artificial intelligence processors, neural network processors, neural processing units (NPUs), artificial intelligence accelerators, application-specific integrated circuits (ASICs), analog signal processors, analog computers, microprocessors, digital signal processors, field-programmable gate arrays (FPGAs), tensor processing units (TPUs), data processing units (DPUs), etc.

[0107] Computing device 1300 may include memory 1304, which itself may include one or more storage devices such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard drives. Memory 1304 includes one or more non-transitory computer-readable storage media. In some embodiments, memory 1304 may include memory that shares a wafer with processing device 1302.

[0108] In some embodiments, memory 1304 includes one or more non-transitory computer-readable media storing executable instructions to perform the operations illustrated in the figures and described herein, such as the methods and operations shown in the figures. In some embodiments, memory 1304 includes storing executable instructions to perform... Figure 10 Method 1000 operation Figure 11 Method 1100 operation, and Figure 12 Method 1200 operates on one or more non-transitory computer-readable media. Exemplary components are described that can be encoded into instructions and stored in memory 1304. Memory 1304 can store instructions encoded on one or more exemplary components, such as cache controller 802. Instructions stored in one or more non-transitory computer-readable media can be executed by processing device 1302.

[0109] In some implementations, memory 1304 may store data as shown in the figures and described herein, such as data structures, binary data, bits, metadata, files, binary large objects, etc. For example, memory 1304 may store key terms and value terms. Memory 1304 may store surrogate key terms and surrogate value terms. Memory 1304 may store term importance scores. Memory 1304 may store the salience scores of attention heads.

[0110] In some embodiments, computing device 1300 may include communication device 1312 (e.g., one or more communication devices). For example, communication device 1312 may be configured to manage wired and / or wireless communications to transmit data to and from computing device 1300. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that communicate data via modulated electromagnetic radiation over a non-solid medium. Although in some embodiments the associated device may not contain arbitrary wires, the term does not imply that the associated device does not contain arbitrary wires. Communication device 1312 may implement any number of wireless standards or protocols, including but not limited to IEEE standards (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), the Long Term Evolution (LTE) project and any amendments, updates, and / or revisions thereof (e.g., the Advanced LTE project, the Ultra Mobile Broadband (UMB) project (also known as "3GPP2"), etc.). IEEE 802.16, compatible with Broadband Wireless Access (BWA) networks, is commonly referred to as WiMAX, an abbreviation for Global Interoperability for Microwave Access. WiMAX is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. Communication equipment 1312 can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication equipment 1312 can operate according to GSM Evolution Enhanced Data (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). Communication equipment 1312 can operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Wireless Communication (DECT), Evolved Data Optimized (EV-DO), and their derivatives, as well as any other wireless protocols designed for 3G, 4G, 5G, etc. In other embodiments, communication device 1312 may operate according to other wireless protocols. Computing device 1300 may include an antenna 1322 to facilitate wireless communication and / or receiving other wireless communications (such as radio frequency transmissions). Computing device 1300 may include receiver circuitry and / or transmitter circuitry. In some embodiments, communication device 1312 may manage wired communication, such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, communication device 1312 may include multiple communication chips.For example, the first communication device 1312 may be dedicated to short-range wireless communication, such as Wi-Fi or Bluetooth, and the second communication device 1312 may be dedicated to long-range wireless communication, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, the first communication device 1312 may be dedicated to wireless communication, and the second communication device 1312 may be dedicated to wired communication.

[0111] The computing device 1300 may include a power supply / power circuit 1314. The power supply / power circuit 1314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuits for coupling components of the computing device 1300 to a power source (e.g., DC power, AC power, etc.) that is separate from the computing device 1300.

[0112] The computing device 1300 may include a display device 1306 (or a corresponding interface circuit as discussed above). The display device 1306 may include any visual indicator, such as, for example, a head-mounted display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0113] The computing device 1300 may include an audio output device 1308 (or a corresponding interface circuit as discussed above). The audio output device 1308 may include any device that generates an audio indicator, such as, for example, a speaker, headphones, or earphones.

[0114] The computing device 1300 may include an audio input device 1318 (or a corresponding interface circuit as discussed above). The audio input device 1318 may include any device that generates a signal representation of sound, such as a microphone, microphone array, or digital instrument (e.g., an instrument with a Musical Instrument Digital Interface (MIDI) output).

[0115] The computing device 1300 may include a GPS device 1316 (or a corresponding interface circuit as discussed above). As is known in the art, the GPS device 1316 can communicate with a satellite-based system and can receive the location of the computing device 1300.

[0116] Computing device 1300 may include sensor 1330 (or one or more sensors). Computing device 1300 may include corresponding interface circuitry as discussed above. Sensor 1330 can sense physical phenomena and convert them into processed electrical signals, such as electrical signals processed by processing device 1302. Embodiments of sensor 1330 may include: capacitive sensors, inductive sensors, resistive sensors, electromagnetic field sensors, light sensors, cameras, imagers, microphones, pressure sensors, temperature sensors, vibration sensors, accelerometers, gyroscopes, stress sensors, humidity sensors, distance sensors, range sensors, time-of-flight sensors, pH sensors, particle sensors, air quality sensors, chemical sensors, gas sensors, biosensors, ultrasonic sensors, scanners, etc.

[0117] The computing device 1300 may include another output device 1310 (or a corresponding interface circuit as discussed above). Embodiments of the other output device 1310 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a haptic output device, a gas output device, a vibrating output device, a light output device, a home automation controller, or additional storage devices.

[0118] The computing device 1300 may include another input device 1320 (or a corresponding interface circuit as discussed above). Embodiments of the other input device 1320 may include an accelerometer, gyroscope, compass, image capture device, keyboard, cursor control device, such as a mouse, stylus, touchpad, barcode reader, quick-response (QR) code reader, arbitrary sensor, or radio frequency identification (RFID) reader.

[0119] The computing device 1300 can have any desired form factor, such as a handheld or mobile computer system (e.g., a cellular phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, personal digital assistant (PDA), personal computer, remote control, wearable device, head-mounted device, glasses, footwear, electronic clothing, etc.), a desktop computer system, a server or other interconnected computing component, a printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, Internet of Things, or wearable computer system. In some embodiments, the computing device 1300 can be any other electronic device that processes data. Select an embodiment

[0120] Example 1 provides an apparatus comprising: at least one computer processor; and one or more memories storing data and instructions at different clustering levels; wherein, when the instructions are executed, the at least one computer processor: determines pruning of lexical units based on their importance, the lexical units being part of a request for a neural network; calculates a saliency score of an attention head of the neural network; selects a clustering level of the attention head from the different clustering levels based on the saliency score; stores surrogate key tensors and surrogate value tensors generated at the clustering levels in a memory of the one or more memories, the surrogate key tensors and surrogate value tensors representing key tensors and value tensors calculated by the attention head of the lexical units; and provides the surrogate key tensors and surrogate value tensors to computational logic performing one or more operations of the attention head.

[0121] Example 2 provides the apparatus of Example 1, wherein the surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor.

[0122] Example 3 provides the apparatus of Example 1 or 2, wherein at least one computer processor further: determines that another term is important and that the other term is another part of the request; stores the other key tensor and the other value tensor computed by the attention head of the other term in another memory of one or more memories; and provides the other key tensor and the other value tensor to the computation logic.

[0123] Example 4 provides an apparatus for any of Examples 1-3, wherein determining to prune a lexical includes comparing the attention weight corresponding to the lexical with a threshold.

[0124] Example 5 provides an apparatus for any of Examples 1-4, wherein calculating the saliency score of the attention head includes: calculating the cosine similarity between the input of the attention head and the output of the attention head.

[0125] Example 6 provides an apparatus for any of Examples 1-5, wherein different clustering levels correspond to different ranges of significance scores.

[0126] Example 7 provides an apparatus of any one of Examples 1-6, wherein the cluster level corresponds to a subrange of significance scores; and determining the cluster level includes determining that the significance score falls within a subrange of significance scores.

[0127] Example 8 provides an apparatus of any one of Examples 1-7, wherein at least one computer processor further: clusters one or more key tensors and one or more value tensors computed by the attention head according to different clustering levels; and determines surrogate key tensors and surrogate value tensors based on one or more centroids at the clustering levels of the one or more clusters.

[0128] Example 9 provides one or more non-transitory computer-readable media storing processor-executable instructions to perform operations for memory management, including: pruning lexical terms based on importance determination, the lexical terms being part of a request for a neural network; calculating a saliency score of an attention head of the neural network; selecting a clustering level of the attention head from different clustering levels based on the saliency score; storing surrogate key tensors and surrogate value tensors generated at the clustering levels in one of the one or more memories, the surrogate key tensors and surrogate value tensors representing key tensors and value tensors calculated by the attention head of the lexical terms, and the one or more memories storing data at different clustering levels; and providing the surrogate key tensors and surrogate value tensors to computational logic performing one or more operations of the attention head.

[0129] Example 10 provides one or more non-transitory computer-readable media of Example 9, wherein the surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor.

[0130] Example 11 provides one or more non-transitory computer-readable media of Example 9 or 10, wherein the operation further includes: determining that another term is important and that the other term is another part of the request; storing another key tensor and another value tensor computed by the attention head in another memory of one or more memories; and providing the other key tensor and the other value tensor to the computation logic.

[0131] Example 12 provides one or more non-transitory computer-readable media of any of Examples 9-11, wherein determining to prune lexical units includes comparing attention weights corresponding to lexical units with thresholds.

[0132] Example 13 provides one or more non-transitory computer-readable media of any of Examples 9-12, wherein calculating the saliency score of the attention head includes: calculating the cosine similarity between the input and output of the attention head.

[0133] Example 14 provides one or more non-transitory computer-readable media of any of Examples 9-13, wherein different clustering levels correspond to different ranges of significance scores.

[0134] Example 15 provides one or more non-transitory computer-readable media of any of Examples 9-14, wherein a cluster level corresponds to a subrange of significance scores; and determining a cluster level includes determining that a significance score falls within a subrange of significance scores.

[0135] Example 16 provides one or more non-transitory computer-readable media of any of Examples 9-15, wherein the operation further includes: clustering one or more key tensors and one or more value tensors according to different clustering levels; and determining surrogate key tensors and surrogate value tensors based on one or more centroids at the clustering levels of the one or more clusters.

[0136] Example 17 provides a method comprising: determining pruning of lexical units based on their importance, the lexical units being part of a request for a neural network; calculating a saliency score of an attention head of the neural network; selecting a clustering level of the attention head from different clustering levels based on the saliency score; storing surrogate key tensors and surrogate value tensors generated at the clustering levels in one of one or more memories, the surrogate key tensors and surrogate value tensors representing key tensors and value tensors computed by the attention head of the lexical units, and the one or more memories storing data at different clustering levels; and providing the surrogate key tensors and surrogate value tensors to computational logic performing one or more operations of the attention head.

[0137] Example 18 provides the method of Example 17, where the surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor, respectively.

[0138] Example 19 provides the method of Example 17 or 18, further comprising: determining that another term is important and that the other term is another part of the request; storing the other key tensor and the other value tensor computed by the attention head in another memory of one or more memories; and providing the other key tensor and the other value tensor to the computation logic.

[0139] Example 20 provides a method from any of Examples 17-19, wherein determining to prune a lexical includes comparing the attention weight corresponding to the lexical with a threshold.

[0140] Example 21 provides a method from any of Examples 17-20, wherein calculating the saliency score of the attention head includes: calculating the cosine similarity between the input and output of the attention head.

[0141] Example 22 provides a method for any of Examples 17-21, where different clustering levels correspond to different ranges of significance scores.

[0142] Example 23 provides a method for any of Examples 17-22, wherein the cluster level corresponds to a subrange of significance scores; and determining the cluster level includes determining that the significance score falls within a subrange of significance scores.

[0143] Example 24 provides a method of any one of Examples 17-23, further comprising: clustering one or more key tensors and one or more value tensors according to different clustering levels; and determining surrogate key tensors and surrogate value tensors based on one or more centroids of the one or more clusters at the clustering level.

[0144] Example A includes an apparatus that includes components that perform any of the methods in Examples 17-24.

[0145] Instance B includes a cache controller as described herein.

[0146] Instance C includes a computing system having, as shown here (such as...) Figure 8 The computing kernel, memory, and cache controller described in the text. Variations and other instructions

[0147] although Figure 10-12 Shown and referenced Figure 10-12 The operations of the described exemplary method are shown to occur once for each operation and in a specific order. It will be appreciated that the operations can be performed in any suitable order and repeated as needed. Furthermore, one or more operations can be performed in parallel. Further, Figure 10-12 The operations shown may be combined or may include more or less detail than described.

[0148] The various implementations described here can be referred to as artificial intelligence, machine learning, and deep learning. Deep learning can be a subset of machine learning. Machine learning can be a subset of artificial intelligence. When referring to deep learning models, machine learning models may be used instead if appropriate for a specific application. Digital signal processing systems may be used instead of deep learning models if appropriate for a specific application.

[0149] The foregoing description, including the illustrated implementations of this disclosure as described in the abstract, is not intended to be exhaustive or to limit this disclosure to the precise form disclosed. Those skilled in the art will recognize that, although specific aspects and embodiments of this disclosure have been described herein for illustrative purposes, various equivalent variations may be possible within the scope of this disclosure. These variations can be made to this disclosure based on the detailed description above.

[0150] For illustrative purposes, specific quantities, materials, and configurations are specified to provide a comprehensive understanding of the exemplary implementation. However, those skilled in the art will recognize that this disclosure can be practiced without specific details and / or only with some of the aspects described. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrated implementation.

[0151] Furthermore, embodiments can be practiced with reference to the accompanying drawings, which form part of this specification and are shown therein by way of illustration. It will be understood that other embodiments can be utilized, and structural or logical changes can be made without departing from the scope of this disclosure. Therefore, the following detailed description is not to be considered limiting.

[0152] Each operation can instead be described as a plurality of discrete actions or operations in a manner most helpful in understanding the disclosed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially dependent. Specifically, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described implementation. In additional implementations, various additional operations may be performed or the described operations may be omitted.

[0153] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" refers to (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" refers to (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used in reference to a strategy range, the term "between" includes the end of the measurement range.

[0154] The description uses the phrases "in one embodiment" or "in an embodiment," each of which may refer to one or more of the same or different embodiments. As used with reference to embodiments of this disclosure, the terms "comprising," "including," "having," etc., are synonymous. This disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to illustrate various features of the drawings; however, these terms are for convenience of discussion only and do not imply any desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, ordinal adjectives such as "first," "second," and "third," etc., used to describe common objects, merely indicate that different instances of similar objects are referred to and are not intended to imply that the described objects must be in a given sequence in time, space, order, or any other manner.

[0155] In the following detailed description, terms commonly used by those skilled in the art will be used to describe various aspects of the illustrated implementations in order to convey the substance of their function to those skilled in the art.

[0156] The terms “substantially,” “near,” “approximately,” “near,” and “about” generally refer to within + / - 20% of the target value described herein or known in the art. Similarly, terms indicating the orientation of individual elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to within + / - 5-20% of the target value described herein or known in the art.

[0157] Furthermore, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to the method, process, or apparatus. Additionally, the term “or” means to include “or” and does not exclude “or.”

[0158] The systems, methods, and apparatus of this disclosure each have several innovative aspects, and no single aspect alone is solely responsible for all the desired properties described herein. Details of one or more implementations of the subject matter described herein are specified in the description and accompanying drawings.

Claims

1. An apparatus comprising: At least one computer processor; as well as One or more memories store data and instructions at different cluster levels; Wherein, when the instructions are executed, the at least one computer processor: The lexical is pruned based on its importance, and the lexical is part of a request to the neural network. Calculate the saliency score of the attention head of the neural network; The clustering level of the attention head is selected from different clustering levels based on the significance score; The surrogate key tensors and surrogate value tensors generated at the clustering level are stored in one of the one or more memories, where the surrogate key tensors and surrogate value tensors represent the key tensors and value tensors calculated by the attention head of the term; and The surrogate key tensor and the surrogate value tensor are provided to computational logic that performs one or more operations on the attention head.

2. The apparatus according to claim 1, wherein, The surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor, respectively.

3. The apparatus according to claim 1 or 2, wherein, The at least one computer processor further includes: It is important to identify another term, which is another part of the request; The other key tensor and the other value tensor calculated by the attention head of the other word are stored in another memory of the one or more memories; and The other key tensor and the other value tensor are provided to the computation logic.

4. The apparatus according to claim 1 or 2, wherein, Determining to trim the lexical units includes: The attention weight corresponding to the lexical is compared with a threshold.

5. The apparatus according to claim 1 or 2, wherein, Calculating the saliency score of the attention head includes: Calculate the cosine similarity between the input and output of the attention head.

6. The apparatus according to claim 1 or 2, wherein, The different clustering levels correspond to different ranges of significance scores.

7. The apparatus according to claim 1 or 2, wherein, The cluster level corresponds to a subrange of significance score; and Determining the cluster level includes determining that the significance score falls within the subrange of the significance score.

8. The apparatus according to claim 1 or 2, wherein, The at least one computer processor further includes: The attention head will cluster one or more key tensors and one or more value tensors according to the different clustering levels; and The surrogate key tensor and the surrogate value tensor are determined based on one or more centroids at the cluster level of one or more clusters.

9. One or more non-transitory computer-readable media storing processor-executable instructions to perform operations for memory management, said operations including: The lexical is pruned based on its importance, and the lexical is part of a request to the neural network. Calculate the saliency score of the attention head of the neural network; The clustering level of the attention head is selected from different clustering levels based on the significance score; The surrogate key tensors and surrogate value tensors generated at the clustering level are stored in one of one or more memories, the surrogate key tensors and surrogate value tensors representing key tensors and value tensors computed by the attention head of the terminology, and the one or more memories store data at the different clustering levels; and The surrogate key tensor and the surrogate value tensor are provided to computational logic that performs one or more operations on the attention head.

10. One or more non-transitory computer-readable media according to claim 9, wherein, The surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor, respectively.

11. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, The operation further includes: It is important to identify another term, which is another part of the request; The other key tensor and another value tensor calculated by the attention head are stored in another memory of the one or more memories; and The other key tensor and the other value tensor are provided to the computation logic.

12. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, Determining to trim the lexical units includes: The attention weight corresponding to the lexical is compared with a threshold.

13. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, Calculating the saliency score of the attention head includes: Calculate the cosine similarity between the input and output of the attention head.

14. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, The different clustering levels correspond to different ranges of significance scores.

15. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, The cluster level corresponds to a subrange of significance score; and Determining the cluster level includes determining that the significance score falls within the subrange of the significance score.

16. One or more non-transitory computer-readable media according to claim 9 or 10, wherein, The operation further includes: Cluster one or more key tensors and one or more value tensors according to the different clustering levels; and The surrogate key tensor and the surrogate value tensor are determined based on one or more centroids at the cluster level of one or more clusters.

17. A method comprising: The lexical is pruned based on its importance, and the lexical is part of a request to the neural network. Calculate the saliency score of the attention head of the neural network; The clustering level of the attention head is selected from different clustering levels based on the significance score; The surrogate key tensors and surrogate value tensors generated at the clustering level are stored in one of one or more memories, the surrogate key tensors and surrogate value tensors representing key tensors and value tensors computed by the attention head of the terminology, and the one or more memories store data at the different clustering levels; and The surrogate key tensor and the surrogate value tensor are provided to computational logic that performs one or more operations on the attention head.

18. The method according to claim 17, wherein, The surrogate key tensor and the surrogate value tensor are approximations of the key tensor and the value tensor, respectively.

19. The method according to claim 17 or 18, further comprising: It is important to identify another term, which is another part of the request; The other key tensor and another value tensor calculated by the attention head are stored in another memory of the one or more memories; and The other key tensor and the other value tensor are provided to the computation logic.

20. The method according to claim 17 or 18, wherein, Determining to trim the lexical units includes: The attention weight corresponding to the lexical is compared with a threshold.

21. The method according to claim 17 or 18, wherein, Calculating the saliency score of the attention head includes: Calculate the cosine similarity between the input and output of the attention head.

22. The method according to claim 17 or 18, wherein, The different clustering levels correspond to different ranges of significance scores.

23. The method according to claim 17 or 18, wherein, The cluster level corresponds to a subrange of significance score; and Determining the cluster level includes determining that the significance score falls within the subrange of the significance score.

24. The method of claim 17 or 18, further comprising: Cluster one or more key tensors and one or more value tensors according to the different clustering levels; and The surrogate key tensor and the surrogate value tensor are determined based on one or more centroids at the cluster level of one or more clusters.

Images