A method, system, device, and medium for KV cache compression of large language models based on retrieval enhancement.
By recognizing generation patterns in a large language model and combining GPU and CPU-side retrieval with an adaptive KV caching compression method, the problem of the imbalance between accuracy and efficiency in existing technologies is solved, and efficient long text generation is achieved.
Patent Information
- Application Number
- CN202611233509.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing key-value caching optimization methods for large language models struggle to balance accuracy and efficiency, and lack adaptive compression strategies, resulting in significant accuracy loss and inference latency in long text generation tasks.
By identifying the generation pattern before inference begins, the optimal compression strategy is automatically selected. Combining GPU memory and CPU-side retrieval, adaptive KV cache compression and asynchronous parallel processing are adopted to achieve efficient utilization of the KV cache.
With a total key-value budget of 15% of the total number of original context key-value pairs, it achieves near-lossless inference accuracy, significantly improves the efficiency and accuracy of long text generation, reduces computational overhead, and maintains good versatility and ease of use.
Smart Images

Figure CN123115200A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model reasoning acceleration and memory optimization technology, and in particular to a method, system, device and medium for large language model KV cache compression based on retrieval enhancement. Background Technology
[0002] In recent years, Large Language Models (LLMs) have made significant progress in long text generation scenarios such as multi-turn dialogue, code understanding, and document summarization, with model context lengths expanding from thousands to millions of words. As context length increases, the GPU memory usage of the Key-Value Cache grows linearly, becoming a major bottleneck limiting long-context inference. For example, with the PaLM-540B model, in an inference scenario with a batch size of 512 and a context length of 2048, the Key-Value Cache consumes as much as 3TB of GPU memory, three times the number of model parameters.
[0003] Existing technologies for optimizing KV caching mainly fall into two categories: (1) Compression-based methods, which reduce GPU memory usage by permanently discarding low-importance KV pairs. Representative methods include SnapKV, PyramidKV, H2O (Heavy-HitterOracle), and SqueezeAttention. These methods have high compression ratios and fast inference speeds, but because the discarded KV pairs cannot be accessed again by subsequent tokens, there is a significant loss of accuracy in long generation tasks. (2) Retrieval-based methods, which offload the entire KV cache to CPU memory and retrieve the important KV pairs required for each decoding step through vector retrieval. Representative methods include RetrievalAttention and PQCache (product quantization). These methods are almost lossless in terms of accuracy, but because each decoding step requires synchronous CPU-side retrieval and GPU-CPU I / O transmission, the inference latency increases significantly, limiting their practicality.
[0004] Existing methods struggle to achieve a good balance between accuracy and efficiency. Furthermore, they lack adaptive compression strategies for different generation scenarios (long input, short output; short input, long output; long input, long output) and cannot automatically identify the generation pattern of the current task before inference begins. This results in the selection of compression strategies relying on manual configuration, which lacks flexibility. Summary of the Invention
[0005] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for large language model key-value cache compression based on retrieval enhancement. By identifying the generation pattern of the prompt text before inference begins, the compression strategy can be automatically adapted, and the optimal compression scheme can be selected for different generation scenarios without manual intervention. Through generation pattern recognition, adaptive key-value cache compression, and CPU-side retrieval enhancement, asynchronous parallelism between GPU-side inference and CPU-side retrieval is achieved, which significantly improves the accuracy of long text generation while maintaining inference efficiency.
[0006] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows: A key-value caching compression method for large language models based on retrieval enhancement includes the following steps: Step 1: Identify the generation pattern of the prompt text before inference begins. Based on the identification results, determine the target key-value retention ratio and CPU-side retrieval budget ratio β for the pre-filling stage and the decoding stage, respectively. Step 2: Pre-fill the prompt text entered by the user to generate an initial KV cache; combine the initial KV cache with the KV pairs of new words generated on the GPU side to form historical KV pairs; Step 3: Perform compression based on the generation mode and target KV retention ratio, select the corresponding observation window, perform voting and scoring operations on historical KV pairs, and determine whether the prompt word unit score is Top-k; if so, store the top k KV pairs with the highest scores in GPU memory; if not, unload the eliminated KV pairs to the CPU side vector database, and at the same time construct the vector index of the KV pairs. Step 4: On the GPU side, new lexical units are generated incrementally using an autoregressive approach. Their key-value pairs are calculated and appended to the end of the initial key-value cache. Simultaneously, for the lexical unit to be generated, the query vector corresponding to the lexical unit generated in the previous time step is used as the retrieval vector. A maximum inner product search request is asynchronously submitted to the CPU side. The top kr key-value pairs with the highest scores are retrieved from the key-value pairs unloaded by the CPU side, where kr = min{β×Lcontext, |KE|}, Lcontext is the total number of key-value pairs in the current context, and KE is the set of key-value pairs unloaded by the CPU side. The retrieval results are returned for use by subsequent lexical units. Step 5: After the GPU receives the KV pair returned by the CPU in Step 4, it concatenates it with the compressed KV cache in the GPU memory. The attention score matrix is then calculated using the query vector of the subsequent word to be generated and the concatenated K and V, and the subsequent word is output. Steps 4 and 5 are repeated until the generation is complete.
[0007] In step 1, the generation of pattern recognition specifically involves: First, extract the prompt length feature of the input prompt text P, that is, count the number of prompt words Lp; compare it with the preset long input threshold Th_p; if Lp>Th_p, then the input is marked as long input, otherwise it is marked as short input; predict the number of words Ld to be generated in the output, and compare it with the preset long generation threshold Th_d; if Ld>Th_d, then the output is marked as long output, otherwise it is marked as short output. Subsequently, the task type features of the prompt text P are extracted. The task types include text summarization, code generation, and multi-turn dialogue. Among them, text summarization tasks correspond to a short output preference, and the output is marked as short output; code generation and multi-turn dialogue tasks correspond to a long output preference, and the output is marked as long output. Finally, based on the input and output tags, the generation mode of the current inference task is determined according to the following rules: long input and short output is LISO mode, short input and long output is SILO mode; long input and long output is LILO mode; and short input and short output use full KV caching for regular inference.
[0008] In step 1, the target KV retention ratio is determined as follows: In the case of long input or long output, based on the number of cue tokens Lp, the target KV retention ratio in the pre-filling stage is set to rp = Th_p / Lp; based on the expected number of generated tokens Ld, the target KV retention ratio in the decoding stage is set to rd = Th_d / Ld; the retrieval budget ratio β is a preset value between 0 and 1, used to determine the number of KV pairs recalled in a single CPU session.
[0009] In step 3, the specific process of adaptive KV cache compression is as follows: In LISO mode, after the pre-filling stage is completed, a one-time compression is performed, and all tokens are evenly divided into multiple windows. The last segment of the prompt text is selected as the observation window L3, and all historical key-value pairs are voted and scored. The voting scores are sorted in descending order, and the top k key-value pairs with the highest scores are selected and stored in the GPU memory. The remaining key-value pairs are discarded and unloaded to the vector database maintained on the CPU side. All newly generated key-value pairs in the decoding stage are stored in the GPU memory. In SILO mode, compression is not performed after the pre-filling stage is completed. Instead, all historical KV pairs are stored in the GPU memory. In the decoding stage, group compression is performed, that is, compression is performed once every Ln new words are generated. The KV pairs corresponding to the most recently generated Ln words are selected as the observation window L3, and the generated historical KV pairs are voted and scored. Then, the top k KV pairs are selected from high to low according to the voting scores and retained in the GPU memory. The remaining KV pairs are unloaded to the vector database maintained on the CPU side. In LILO mode, the compression strategy of LISO mode is applied to the KV pairs after the pre-filling stage, and the compression strategy of SILO mode is applied to the KV pairs in the decoding stage.
[0010] In step 3, the construction of the vector index of the KV pair is specifically as follows: In LISO mode, after the pre-filling phase is completed, the discarded KV pairs are offloaded to the CPU-side vector database and a static vector index is created. In SILO mode, during the decoding phase, the first discarded KV pairs are offloaded to the vector database maintained by the CPU and a dynamic vector index is established. Subsequently, each time a new KV pair is discarded during compression, it is incrementally inserted into the existing vector index to achieve dynamic updates. In LILO mode, the pre-filling stage unloads the KV pairs that are compressed and eliminated at once to the vector database maintained by the CPU and establishes an initial dynamic vector index; in the decoding stage, each time a new KV pair is generated by group compression, its increment is inserted into the initial dynamic vector index to achieve dynamic updates.
[0011] In step 3, the specific process of the voting and scoring operation is as follows: Let QL3 be the query matrix corresponding to the observation window, and KL2 be the key matrix in the historical key-value pairs. Calculate the attention score matrix between QL3 and KL2, and sum along the L3 dimension to obtain the voting score SL2 for each historical word. The formula for the attention score matrix is as follows: Where T represents transpose and D represents the dimension per head.
[0012] In step 4, the Maximum Inner Product Search (MIPS) is transformed into the Nearest Neighbor Search (NNS) problem to achieve efficient retrieval. The specific process is as follows: Given a query vector q ∈ R D With FAISS database X R D The MIPS request for maximum inner product search is to find x* = argmax_{x∈X}.<q, x> The MIPS protocol is transformed into NNS using the following transformation: definition , ,in, After transformation ,because and Both are constants; minimizing the Euclidean distance is equivalent to maximizing the inner product. This transforms MIPS into a standard NNS problem.
[0013] This invention also provides a large language model key-value caching and compression system based on retrieval enhancement, comprising: Compression Decision Module: Used to identify the generation pattern of the prompt text before inference begins, and based on the identification results, determine the target key-value retention ratio and CPU-side retrieval budget ratio β for the pre-filling stage and the decoding stage respectively; Pre-filling module: pre-fills the prompt text input by the user to generate an initial KV cache; the initial KV cache and the KV pairs of new words generated on the GPU side are used to form historical KV pairs; Compression module: Performs compression based on generation mode and target KV retention ratio, selects corresponding observation window, performs voting and scoring operation on historical KV pairs, and determines whether the prompt word byte score is Top-k; if so, stores the top k KV pairs with the highest scores in GPU memory; if not, the eliminated KV pairs are unloaded to the CPU side vector database, and a vector index of the KV pairs is constructed at the same time. The retrieval module: On the GPU side, new lexical units are generated incrementally using an autoregressive approach. Their key-value pairs are calculated and appended to the end of the initial key-value cache. Simultaneously, for the lexical unit to be generated, the query vector corresponding to the lexical unit generated in the previous time step is used as the retrieval vector. A maximum inner product search request is asynchronously submitted to the CPU side. The top kr key-value pairs with the highest scores are retrieved from the key-value pairs unloaded from the CPU side, where kr = min{β × Lcontext, |KE|}, Lcontext is the total number of key-value pairs in the current context, and KE is the set of key-value pairs unloaded from the CPU side. The retrieved results are then used for subsequent lexical units. Attention Correction Module: When the GPU receives the KV pair returned by the CPU in step 4, it concatenates it with the compressed KV cache in the GPU memory. The attention score matrix is calculated using the query vector of the subsequent words to be generated and the concatenated K and V, and the subsequent words are output. The retrieval and attention correction are repeated until the generation is completed.
[0014] This invention also provides a retrieval-enhanced large language model key-value caching and compression device, comprising: Memory: A computer program that stores the above-mentioned large language model KV cache compression method, and is a computer-readable device; Processor: Used to implement the large language model KV cache compression method described above when executing the computer program.
[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described large language model KV cache compression method.
[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention uses a generation pattern recognizer to automatically analyze the length and task type characteristics of the input prompt text before inference begins, and determines the current inference task as one of the three generation modes: LISO, SILO, or LILO. Based on this, it automatically selects the corresponding KV caching compression strategy, which does not require manual configuration by the user and significantly improves the ease of use and applicability of the framework.
[0017] 2. This invention combines a key-value (KV) cache compression method with a CPU-side retrieval enhancement method. By retaining a subset of compressed KV cache in GPU memory, it stores obsolete KV pairs in a vector database maintained on the CPU side and retrieves historical KV pairs that are important for subsequent word generation through asynchronous MIPS retrieval. It achieves near-lossless inference accuracy with a total KV budget of 15% of the total original context KV pairs, resulting in a relative accuracy improvement of approximately 2.9% to 9.6% compared to existing compression methods.
[0018] 3. This invention adopts an asynchronous retrieval strategy that uses the query vector of the generated tokens as the retrieval agent. It takes advantage of the high similarity of the attention allocation patterns of adjacent tokens to complete the CPU-side retrieval without blocking the GPU inference. The inference latency is comparable to that of pure compression methods and is much lower than that of existing retrieval methods.
[0019] 4. The group compression strategy proposed in this invention performs a compression operation only once for every Ln new tokens generated, which greatly reduces the frequency of compression operations and reduces additional computational overhead. At the same time, since the attention patterns of adjacent tokens are similar, group compression will not cause a significant decrease in the quality of KV cache.
[0020] 5. The retrieval module of this invention is orthogonally compatible with existing KV cache compression methods such as SnapKV and PyramidKV, without changing their original compression strategies. It only unloads the eliminated KV pairs to the CPU side to establish an index, and asynchronously retrieves the relevant KV pairs during the decoding stage. After being concatenated with the KV cache retained on the GPU side, it participates in attention calculation, thereby recalling important information that was mistakenly eliminated and reducing the loss of compression accuracy. It has good versatility and scalability.
[0021] In summary, this invention automatically analyzes the length features, task type features, and expected number of generated tokens of the input prompt text before inference begins using a generative pattern recognizer. It adaptively determines the inference task into one of three generation modes: Long Input Short Output (LISO), Short Input Long Output (SILO), or Long Input Long Output (LILO). For short input and short output, a full key-value (KV) cache is used, and the corresponding KV cache compression strategy is selected accordingly, eliminating the need for manual configuration and improving the framework's usability. At the compression strategy level, this invention maintains a high-quality KV cache subset in GPU memory through voting and scoring, while simultaneously offloading discarded KV pairs to a CPU-side vector database and constructing a vector index. During the decoding stage, the query vector of the generated tokens is used as a retrieval proxy to asynchronously submit a Maximum Inner Product Search (MIPS) request to the CPU. The retrieval result is concatenated with the GPU-side compressed KV cache and used for attention correction of subsequent tokens, achieving near-lossless inference accuracy with a total KV budget of only 15% of the original context KV pairs. The system, device, and medium are used to implement this method. This invention achieves a good balance between accuracy, efficiency, and memory usage. Its inference accuracy on LongGenBench and LongProc benchmarks is significantly better than existing KV compression methods, while its generation latency is comparable to that of pure compression methods. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the framework of the large language model KV cache compression system based on retrieval enhancement provided by the present invention.
[0023] Figure 2 This is a flowchart illustrating the KV cache compression method for a large language model based on retrieval enhancement provided by the present invention.
[0024] Figure 3 Flowchart of compression operation Figure 4 This is a flowchart illustrating the retrieval process.
[0025] Figure 5 This is a schematic diagram illustrating the observation window voting mechanism in three scenarios; among them, Figure 5 (a) is the LISO scene. Figure 5 (b) is the SILO scenario. Figure 5 (c) is the LILO scene.
[0026] Figure 6 This paper compares the completion rates of the compression method of this invention with those of existing methods on the Longgenbench dataset.
[0027] Figure 7 This paper compares the accuracy of the compression method of the present invention with existing methods on the Longgenbench dataset.
[0028] Figure 8a This is a comparison of the initial character generation delay between the present invention and existing methods.
[0029] Figure 8b This is a comparison of the continuous generation speed of the present invention and existing methods.
[0030] Figure 9 This is a comparison of the relative accuracy improvement rate of the retrieval module of this invention when combined with SnapKV and PyramidKV. Detailed Implementation
[0031] The technical solution of the present invention will be further described in detail with reference to the accompanying drawings.
[0032] like Figure 1 As shown, this invention proposes a retrieval-enhanced key-value (KV) caching and compression system for large language models, comprising a compression decision module, a pre-filling module, a compression module, a retrieval module, and an attention correction module. Through asynchronous parallelism between GPU-side inference and CPU-side retrieval, it significantly improves the accuracy of long text generation while maintaining inference efficiency. Compression decision module: used to generate pattern recognition of the prompt text before inference begins. Based on the recognition results, the target key-value retention ratio and CPU-side retrieval budget ratio β are determined for the pre-filling stage and the decoding stage respectively. When the input is short and the output is short, no compression is performed, i.e., normal generation is performed. Pre-filling module: pre-fills the prompt text input by the user to generate initial KV pairs; the initial KV buffer and the KV pairs of new words generated on the GPU side are used to form historical KV pairs; Compression module: Based on the generation mode and the target key-value retention ratio, it maintains a compressed key-value cache subset in the GPU memory and offloads the discarded key-value pairs to the CPU side vector database; The retrieval module is used to maintain the vector index on the CPU side, determine the number of retrievals per retrieval kr according to the retrieval budget ratio β, receive asynchronous maximum inner product search (MIPS) requests from the GPU side, retrieve the top kr KV pairs with the highest scores from the unloaded KV pairs and return them to the GPU side; the retrieval results are used by subsequent terms after they are returned.
[0033] Attention Correction Module: After receiving the KV pairs returned from the CPU side on the GPU side, it concatenates the retrieved KV pairs with the compressed KV cache in the GPU memory, performs attention calculations, and outputs subsequent tokens.
[0034] like Figure 2As shown, a retrieval-Augmented KV Cache Compression (RACC) method for large language models includes five main stages: compression decision generation, pre-filling stage processing, adaptive KV cache compression, asynchronous CPU-side retrieval enhancement, attention correction, and output. Its core idea is as follows: before inference, a generative pattern recognizer automatically determines the generation scenario; a compressed subset of highly important KV caches is maintained in GPU memory; simultaneously, KV pairs that are eliminated from compression are unloaded to the CPU-side vector database; and important historical KV pairs are retrieved using asynchronous maximum inner product search (MIPS) without blocking GPU inference. The retrieved historical KV pairs are then concatenated with the compressed KV cache for subsequent attention correction of lexical units. This achieves near-lossless inference accuracy with a total KV budget of only 15% of the original context KV pairs. The specific steps are as follows: Step 1, Compression Decision Generation: Before inference begins, identify the generation pattern of the prompt text. Based on the identification results, determine the target KV retention ratio and CPU-side retrieval budget ratio β for the pre-filling stage and decoding stage respectively. If the input is short and the output is short, no compression is performed, i.e., normal generation is performed. Step 1.1, Generate Pattern Recognition: Before GPU inference begins, the Generator ModeRecognizer performs static analysis on the user-input prompt text P; First, the recognizer extracts the prompt length feature of the input prompt text P, that is, counts the number of prompt tokens Lp, and compares it with a preset long input threshold Th_p; if Lp > Th_p, the input is marked as a long input, otherwise it is marked as a short input; the long input threshold Th_p = Nmaxtoken × ρ, N max `token` represents the total token budget, indicating the maximum number of tokens that the model can input and generate in the current request. It is set by the user. `ρ` represents the token budget weight of the input, which is a constant between 0 and 1.
[0035] The recognizer predicts the number of tokens Ld to be generated and compares it with a preset long generation threshold Th_d. If Ld > Th_d, the output is marked as a long output; otherwise, it is marked as a short output. Where Th_d = N max token×(1-ρ), where 1-ρ represents the generated token budget weight.
[0036] Subsequently, the recognizer extracts the task type feature vector ft from the prompt text P. The task types include text summarization, code generation, and multi-turn dialogue. A lightweight rule set or task type classifier is used to determine whether the current task belongs to the text summarization, code generation, or multi-turn dialogue category. Text summarization tasks usually correspond to shorter outputs, so the output is marked as short output. Code generation and multi-turn dialogue tasks usually correspond to longer outputs, so the output is marked as long output. The output marking is used to assist in predicting the expected number of generated tokens Ld.
[0037] Finally, based on the input and output tags, the recognition module identifies the generation pattern of the current inference task and passes the identified generation pattern result to the compression module for subsequent adaptive strategy selection.
[0038] The generation mode is determined according to the following rules: if the input is labeled as long input and the output is labeled as short output, it is determined to be LISO mode; if the input is labeled as short input and the output is labeled as long output, it is determined to be SILO mode; if the input is labeled as long input and the output is labeled as long output, it is determined to be LILO mode; if the input is labeled as short input and the output is labeled as short output, regular inference is performed using a full KV cache.
[0039] Step 1.2: Determine the target KV retention ratio and retrieval budget ratio β for the pre-filling and decoding stages: When the number of prompt tokens Lp is not greater than the long input threshold Th_p, the target retention ratio rp for the pre-filling stage is 1; otherwise, rp = Th_p / Lp. When the expected number of generated tokens Ld is not greater than the long output threshold Th_d, the target retention ratio rd on the GPU side for the decoding stage is 1; otherwise, rd = Th_d / Ld. Here, rp is used to determine the k value in the Top-k of the pre-filling stage, and rd is used to determine the single retention ratio α and the corresponding k value for group compression in the decoding stage. The retrieval budget ratio β is also defined, representing the proportion of KV pairs recalled from the CPU side to the total number of current context KV pairs.
[0040] Step 2, Pre-filling stage processing: Pre-fill the user-input prompt text to generate an initial KV buffer; specifically: embed the user-input prompt text P into a word vector, add positional encoding, and obtain the query matrix Q_prompt ∈ R through linear projection. {B×S×H×D} The key matrix K_prompt ∈ R {B×S×H×D} Value matrix V_prompt ∈ R {B×S×H×D}Where B is the batch size, S is the sequence length of the prompt text, H is the number of attention heads, D is the dimension per head, and R is the vector set; after completing the forward propagation of the multi-layer Transformer, the key matrix K_prompt and value matrix V_prompt of all layers are stored as the initial KV buffer for all tokens and stored in GPU memory for the generation of the first prompt token in the decoding stage. The initial KV buffer and the KV pairs of the new tokens generated on the GPU side constitute the history KV pairs.
[0041] The formula for the forward propagation of the multi-layer Transformer is as follows: In the formula, T represents the matrix transpose operation; Step 3: Adaptive KV Cache Compression and Online KV Index Construction: The compression module performs compression according to the generation mode and compression ratio determined in Step 1. Specifically, it selects the corresponding observation window, performs a voting and scoring operation on historical KV pairs, and determines whether the prompt word's score is Top-k. If so, the top k KV pairs with the highest scores are stored in GPU memory; otherwise, the discarded KV pairs are unloaded into the CPU side vector database, and a vector index for that KV pair is constructed to support subsequent retrieval. Figure 3 As shown; In LISO mode, after the pre-filling stage is completed, a one-time compression is performed. All tokens are evenly divided into multiple windows. The last segment of the prompt text is selected as the observation window L3. All historical key-value pairs, which are observed windows L2, are voted and scored. The voting scores are sorted in descending order, and the top k key-value pairs with the highest scores are selected and stored in the GPU memory. The remaining key-value pairs are discarded and unloaded to the vector database maintained on the CPU side. All newly generated key-value pairs in the decoding stage are stored in the GPU memory. If the context is short, no compression is performed, i.e., normal generation is performed.
[0042] In SILO mode, compression is not performed after the pre-filling stage; instead, all historical key-value pairs are stored in GPU memory. During the decoding stage, group compression is performed, meaning compression is performed once every Ln new tokens generated. Specifically, the Ln most recently generated key-value pairs are selected as the observation window L3, and a voting and scoring operation is performed on the generated historical key-value pairs. Then, the top k key-value pairs are selected from highest to lowest voting scores and retained in GPU memory, while the remaining key-value pairs are appended and unloaded to the vector database maintained on the CPU side.
[0043] In LILO mode, compression decisions in LISO mode are applied to KV pairs after the pre-filling stage, and compression decisions in SILO mode are applied to KV pairs in the decoding stage.
[0044] During the decoding phase, when the target KV retention ratio rd = 1, no KV cache compression is performed; when rd < 1, group compression is performed once for every Ln new tokens generated. Let Lhistory be the length of the historical KV cache already retained on the GPU side before this compression. After adding the newly generated Ln KV pairs, a voting score is performed on a total of (Lhistory + Ln) KV pairs, and the top k KV pairs are selected from highest to lowest score for retention, where k = (Lhistory + Ln) × α, α is the retention ratio for a single group compression, and the compressed KV cache length Lcompression = k. Let n = Ld / Ln be the expected number of group compressions, then α is set according to the GPU-side target KV retention ratio rd determined in step 1.2. Considering the KV pairs retrieved on the CPU side, the overall KV retention ratio PR satisfies the following relationship: Where rd represents the target KV retention ratio on the GPU side during the decoding stage, β represents the retrieval budget ratio on the CPU side, PR represents the overall KV retention ratio composed of the KV pairs retained on the GPU side and the KV pairs recalled on the CPU side; α is used to control the retention ratio for each group compression, and k represents the number of KV pairs actually retained in the current compression operation.
[0045] The online key-value index construction process is as follows: In LISO mode, after the pre-filling stage is completed, the KV pairs are compressed once, the discarded KV pairs are unloaded to the CPU-side vector database, and a static vector index is established; the index is not updated during the decoding stage.
[0046] In SILO mode, the KV pairs are compressed in groups during the decoding stage. The KV pairs that are to be eliminated are first unloaded to the vector database maintained by the CPU side and a dynamic vector index is established. Each time a new KV pair is eliminated during subsequent compression, it is inserted into the existing vector index in an incremental manner to realize the dynamic update of the vector index.
[0047] In LILO mode, the pre-filling stage performs one-time compression on the KV pairs, unloads the KV pairs to be eliminated to the CPU-side vector database, and establishes a dynamic vector index during the first unloading. During the decoding stage, each time a new KV pair is generated by group compression, its increment is inserted into the initial dynamic vector index to achieve continuous index updates.
[0048] The specific process of the voting and scoring operation is as follows: Let the query matrix corresponding to the observation window be QL3, and the key matrix in the historical KV pairs be KL2. Calculate the attention score matrix between QL3 and KL2, and sum along the L3 dimension to obtain the voting score SL2 for each historical word. The formula for the attention score matrix is as follows: Where T represents transpose.
[0049] Step 4, Enhanced Asynchronous CPU-Side Retrieval: During the decoding phase, the GPU side uses an autoregressive approach to gradually generate new lexical units. After calculating the key-value pairs (KV) of the new lexical units, they are appended to the end of the original initial KV cache to form historical KV pairs. Let the total number of current context KV pairs be Lcontext. Based on the retrieval budget ratio β determined in Step 1.2, the number of single retrievals is calculated as kr = min{β × Lcontext, |KE|}, where KE is the set of KV pairs that have been unloaded from the CPU side. For the lexical unit to be generated, the query vector corresponding to the lexical unit generated in the previous time step is used as the retrieval vector. A maximum inner product search request is asynchronously submitted to the CPU side. The top kr KV pairs with the highest scores are retrieved from the KV pairs that have been unloaded from the CPU side, realizing collaborative reasoning between the CPU and GPU. The GPU side does not need to wait for the retrieval results; the retrieval results are returned and used for subsequent lexical units.
[0050] like Figure 1 and Figure 4 As shown, the vector from the GPU is retrieved, and it is determined whether it is a KV cache vector. If so, an index is constructed for it and stored in the CPU-side vector database; otherwise, it is determined to be a query vector. At regular intervals, a retrieval task is performed on the index in the CPU to retrieve KV pairs that are important for the current generation but were mistakenly discarded, as candidate vectors. For the term to be generated at time t, the query vector q corresponding to the term already generated at time t-1 is used. t-1 As a task proxy, it asynchronously submits Maximum Inner Product Search (MIPS) requests to the CPU, retrieving the top kr highest-scoring KV pairs from the unloaded KV pair set KE on the CPU side, forming a high-importance KV cache subset K'E; that is... k1∈K'E, k2∈KE\K'E, we have t-1 , k1>≥ t-1 Let k1 be a vector from the more important KV cache subset K'E, and k2 be a vector from the remaining less important KV cache subsets KE\K'E. After issuing a MIPS retrieval request, the GPU does not need to wait for the result; it continues to calculate the attention score matrix at time t and continues forward inference. The scheduling module ensures asynchronous parallelism between GPU inference and CPU retrieval. After the CPU-side retrieval is completed, the returned KV pairs are transferred to the GPU via GPU-CPU I / O for subsequent attention correction of tokens.
[0051] CPU-side MIPS retrieval requests are transformed into nearest neighbor search (NNS) problems in standard Euclidean space through vector space transformation, and efficient retrieval is achieved by utilizing the Flat index, IVFPQ index, or HNSW index in the FAISS database (Facebook AI Similarity Search).
[0052] The specific process of converting the MIPS retrieval request into an NNS question is as follows: Given a query vector q ∈ R D With FAISS database X R D The MIPS retrieval request is to find x* = argmax_{x∈X}<q, x> ; The MIPS can be transformed into NNS using the following transformation: definition , ,in, After transformation ,because and Both are constants; minimizing the Euclidean distance is equivalent to maximizing the inner product. This transforms MIPS into a standard NNS problem.
[0053] Step 5, Attention Correction and Lexical Output: After the GPU receives the KV pairs returned by the CPU in Step 4, it concatenates them with the compressed KV cache in the GPU memory: K = Concat(KR, K'E), V = Concat(VR, V'E), where KR and VR are compressed KV pairs stored in the GPU memory, and K'E and V'E are KV pairs returned by the CPU. The query vector q to be generated subsequently... t The attention score matrix is calculated with the concatenated K and V, the representation of the subsequent word is output, and the subsequent layer predicts the output word; then steps 4 and 5 are repeated until the generation is complete.
[0054] Experimental verification This invention was experimentally verified on NVIDIA A6000 GPU and Intel Xeon Gold 6240 CPU platforms, using LLaMA-3.1-8B-Instruct and Mistral-7B-v0.2 as the basic language models. It was compared with full cache and compression baseline methods such as SnapKV, PyramidKV, H2O, SqueezeAttention, Scope, and CakeKV on the LongGenBench and LongProc long text generation benchmarks.
[0055] like Figure 6 and Figure 7 As shown, under the same total KV budget (overall KV retention rate of 15%), the RACC method of this invention significantly outperforms the compressed baseline method in both completion rate and accuracy, generating inference accuracy close to that of full KV cache, thus proving the effectiveness of the retrieval module.
[0056] like Figure 8a and Figure 8b As shown, in terms of initial character generation latency and sustained generation speed, RACC's TTFT is 259 ms, lower than compression methods such as SqueezeAttention, Scope, and CakeKV. Based on the overall generation latency of generating 1024 tokens, RACC's sustained generation speed is approximately 31.60 tokens / s, higher than the aforementioned compression methods and close to the 33.34 tokens / s of full KV caching. Compared to retrieval methods PQCache (approximately 10.97 tokens / s) and RetrievalAttention (approximately 5.31 tokens / s), RACC's sustained generation speed is approximately 2.88 times and 5.95 times higher, respectively, indicating that this invention, while introducing retrieval enhancement, still maintains low initial character generation latency and high sustained generation efficiency.
[0057] like Figure 9 As shown, combining the RACC retrieval module with the pre-filled compression methods SnapKV and PyramidKV respectively, improves accuracy compared to the original compression methods on the LLaMA and Mistral model's gov report, multi-news, and qmsum tasks. Specifically, the relative accuracy improvement rate after combining SnapKV and RACC is approximately 3.1%–9.6%, and the relative accuracy improvement rate after combining PyramidKV and RACC is approximately 2.9%–3.5%, demonstrating the universality and scalability of this invention.
[0058] This embodiment uses the SILO mode, with a single-batch compression retention ratio α set to 60%, a retrieval budget ratio β set to 1%, a group window size Ln set to 400, and an overall KV retention ratio PR set to 15%. Sensitivity analysis results show that increasing the retrieval budget ratio β or increasing the single-batch compression retention ratio α can further improve accuracy, at the cost of correspondingly increasing GPU and CPU memory consumption; increasing the group window Ln will improve GPU memory utilization, but the improvement in accuracy is limited.
Claims
1. A large language model KV cache compression method based on retrieval enhancement, characterized in that, Includes the following steps: Step 1: Identify the generation pattern of the prompt text before inference begins. Based on the identification results, determine the target key-value retention ratio and CPU-side retrieval budget ratio β for the pre-filling stage and the decoding stage, respectively. Step 2: Pre-fill the prompt text entered by the user to generate an initial KV cache; combine the initial KV cache with the KV pairs of new words generated on the GPU side to form historical KV pairs; Step 3: Perform compression based on the generation mode and target KV retention ratio, select the corresponding observation window, perform voting and scoring operations on historical KV pairs, and determine whether the prompt word unit score is Top-k; if so, store the top k KV pairs with the highest scores in GPU memory; if not, unload the eliminated KV pairs to the CPU side vector database, and at the same time construct the vector index of the KV pairs. Step 4: On the GPU side, new lexical units are generated incrementally using an autoregressive approach. Their key-value pairs are calculated and appended to the end of the initial key-value cache. Simultaneously, for the lexical unit to be generated, the query vector corresponding to the lexical unit generated in the previous time step is used as the retrieval vector. A maximum inner product search request is asynchronously submitted to the CPU side. The top kr key-value pairs with the highest scores are retrieved from the key-value pairs unloaded by the CPU side, where kr = min{β×Lcontext, |KE|}, Lcontext is the total number of key-value pairs in the current context, and KE is the set of key-value pairs unloaded by the CPU side. The retrieval results are returned for use by subsequent lexical units. Step 5: After the GPU receives the KV pair returned by the CPU in Step 4, it concatenates it with the compressed KV cache in the GPU memory. The attention score matrix is then calculated using the query vector of the subsequent word to be generated and the concatenated K and V, and the subsequent word is output. Steps 4 and 5 are repeated until the generation is complete.
2. The large language model KV cache compression method according to claim 1, characterized in that: In step 1, the generation of pattern recognition specifically involves: First, extract the prompt length feature of the input prompt text P, that is, count the number of prompt words Lp; compare it with the preset long input threshold Th_p; if Lp>Th_p, then the input is marked as long input, otherwise it is marked as short input; predict the number of words Ld to be generated in the output, and compare it with the preset long generation threshold Th_d; if Ld>Th_d, then the output is marked as long output, otherwise it is marked as short output. Subsequently, the task type features of the prompt text P are extracted. The task types include text summarization, code generation, and multi-turn dialogue. Among them, text summarization tasks correspond to a short output preference, and the output is marked as short output; code generation and multi-turn dialogue tasks correspond to a long output preference, and the output is marked as long output. Finally, based on the input and output tags, the generation mode of the current inference task is determined according to the following rules: long input and short output is LISO mode, short input and long output is SILO mode; long input and long output is LILO mode; and short input and short output use full KV caching for regular inference.
3. The large language model KV cache compression method of claim 2, characterized in that: In step 1, the target KV retention ratio is determined as follows: In the case of long input or long output, based on the number of cue tokens Lp, the target KV retention ratio in the pre-filling stage is set to rp = Th_p / Lp; based on the expected number of generated tokens Ld, the target KV retention ratio in the decoding stage is set to rd = Th_d / Ld; the retrieval budget ratio β is a preset value between 0 and 1, used to determine the number of KV pairs recalled in a single CPU session.
4. The large language model KV cache compression method of claim 1, wherein: In step 3, the specific process of adaptive KV cache compression is as follows: In LISO mode, after the pre-filling stage is completed, a one-time compression is performed, all lexical units are evenly divided into multiple windows, the last segment of the prompt text is selected as the observation window L3, and all historical KV pairs are voted and scored; the voting scores are sorted in descending order, the top k KV pairs with the highest scores are selected and stored in the GPU memory, and the remaining KV pairs are discarded and unloaded to the vector database maintained on the CPU side. All newly generated key-value pairs during the decoding phase are stored in GPU memory; In SILO mode, compression is not performed after the pre-filling stage is completed. Instead, all historical KV pairs are stored in the GPU memory. In the decoding stage, group compression is performed, that is, compression is performed once every Ln new words are generated. The KV pairs corresponding to the most recently generated Ln words are selected as the observation window L3, and the generated historical KV pairs are voted and scored. Then, the top k KV pairs are selected from high to low according to the voting scores and retained in the GPU memory. The remaining KV pairs are unloaded to the vector database maintained on the CPU side. In LILO mode, the compression strategy of LISO mode is applied to the KV pairs after the pre-filling stage, and the compression strategy of SILO mode is applied to the KV pairs in the decoding stage.
5. The large language model KV cache compression method of claim 4, characterized in that: In step 3, the construction of the vector index of the KV pair is specifically as follows: In LISO mode, after the pre-filling phase is completed, the discarded KV pairs are offloaded to the CPU-side vector database and a static vector index is created. In SILO mode, during the decoding phase, the first discarded KV pairs are offloaded to the vector database maintained by the CPU and a dynamic vector index is established. Subsequently, each time a new KV pair is discarded during compression, it is incrementally inserted into the existing vector index to achieve dynamic updates. In LILO mode, the pre-filling stage unloads the KV pairs that are compressed and eliminated at once to the vector database maintained by the CPU and establishes an initial dynamic vector index; in the decoding stage, each time a new KV pair is generated by group compression, its increment is inserted into the initial dynamic vector index to achieve dynamic updates.
6. The large language model KV cache compression method of claim 4, characterized in that: In step 3, the specific process of the voting and scoring operation is as follows: Let QL3 be the query matrix corresponding to the observation window, and KL2 be the key matrix in the historical key-value pairs. Calculate the attention score matrix between QL3 and KL2, and sum along the L3 dimension to obtain the voting score SL2 for each historical word. The formula for the attention score matrix is as follows: Where T represents transpose and D represents the dimension per head.
7. The large language model KV cache compression method of claim 1, wherein: In step 4, the Maximum Inner Product Search (MIPS) is transformed into the Nearest Neighbor Search (NNS) problem to achieve efficient retrieval. The specific process is as follows: Given a query vector q ∈ R D with the FAISS database X R D The maximum inner product search MIPS request is to find x* = argmax_{x∈X} <q, x>; MIPS is transformed into NNS by the following transformation: definition , ,in, After transformation ,because and Both are constants; minimizing the Euclidean distance is equivalent to maximizing the inner product. This transforms MIPS into a standard NNS problem.
8. A large language model key-value caching and compression system based on retrieval enhancement, characterized in that, include: Compression Decision Module: Used to identify the generation pattern of the prompt text before inference begins, and based on the identification results, determine the target key-value retention ratio and CPU-side retrieval budget ratio β for the pre-filling stage and the decoding stage respectively; Pre-filling module: pre-fills the prompt text input by the user to generate an initial KV cache; the initial KV cache and the KV pairs of new words generated on the GPU side are used to form historical KV pairs; Compression module: Performs compression based on generation mode and target KV retention ratio, selects corresponding observation window, performs voting and scoring operation on historical KV pairs, and determines whether the prompt word byte score is Top-k; if so, stores the top k KV pairs with the highest scores in GPU memory; if not, the eliminated KV pairs are unloaded to the CPU side vector database, and a vector index of the KV pairs is constructed at the same time. The retrieval module: On the GPU side, new lexical units are generated incrementally using an autoregressive approach. Their key-value pairs are calculated and appended to the end of the initial key-value cache. Simultaneously, for the lexical unit to be generated, the query vector corresponding to the lexical unit generated in the previous time step is used as the retrieval vector. A maximum inner product search request is asynchronously submitted to the CPU side. The top kr key-value pairs with the highest scores are retrieved from the key-value pairs unloaded from the CPU side, where kr = min{β × Lcontext, |KE|}, Lcontext is the total number of key-value pairs in the current context, and KE is the set of key-value pairs unloaded from the CPU side. The retrieved results are then used for subsequent lexical units. Attention Correction Module: When the GPU receives the KV pair returned by the CPU in step 4, it concatenates it with the compressed KV cache in the GPU memory, calculates the attention score matrix with the query vector of the subsequent word to be generated and the concatenated K and V, and outputs the subsequent word. Repeat the retrieval and attention adjustment until the generation is complete.
9. A large language model key-value caching and compression device based on retrieval enhancement, characterized in that, include: Memory: A computer program for the large language model KV cache compression method according to any one of claims 1-7, which is a computer-readable device; Processor: Used to implement the large language model KV cache compression method according to any one of claims 1-7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, can implement the large language model KV cache compression method according to any one of claims 1-7.
