Large model KV cache compression method, system, product and equipment
By combining global low-rank approximation and local residual compensation with multi-scale window analysis and adaptive sparse threshold optimization, the storage bottleneck of large model KV cache is solved, achieving a balance between high compression rate and high fidelity, and improving inference speed and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2025-11-13
- Publication Date
- 2026-04-21
AI Technical Summary
Existing technologies suffer from storage bottlenecks in optimizing large model key-value caches, making it difficult to achieve high compression rates, high fidelity, low computational overhead, and strong adaptability, especially in long-sequence inference and resource-constrained devices.
We employ a global low-rank approximation and local residual compensation method. By reducing global storage overhead through low-rank decomposition, we introduce multi-scale window analysis and adaptive sparse threshold optimization to accurately preserve details in high-information-density regions. Combined with dynamic complementary management, we can adapt to different tasks and hardware environments.
It achieves a balance between high compression rate and high fidelity in KV caching, improves inference speed and adaptability, and solves the memory bottleneck problem of large models in long sequences and resource-constrained devices.
Smart Images
Figure CN121900932A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a large model KV cache compression method, system, product, and device based on global low-rank approximation and local residual compensation. Background Technology
[0002] In the inference process of large AI models (especially the Transformer architecture), key-value (KV) caching is a core component for ensuring inference efficiency. Its core function is to store the key matrix K and value matrix V generated during inference, avoiding redundant calculations and improving response speed. However, as large models evolve towards larger parameter numbers and longer sequence lengths, the storage requirements of KV caching exhibit a significant O(Nd) growth characteristic with both sequence length N and model dimension d. When the sequence length N expands to thousands or even tens of thousands of tokens, and the model dimension d reaches thousands of dimensions, the storage overhead of KV caching increases dramatically, becoming a key memory bottleneck restricting the deployment of large models. This problem is particularly prominent in long-sequence inference scenarios (such as document generation, machine translation, and multi-turn dialogue) and resource-constrained devices (such as edge computing devices and mobile terminals), directly limiting the application scope and deployment efficiency of large models.
[0003] To address the storage bottleneck of key-value (KV) caching, various optimization solutions have emerged in the industry, primarily including quantization, sparsity, pruning, attention-based sparsity, multi-head processing, and targeted methods such as CacheGen. The core idea behind these techniques largely revolves around "reducing storage redundancy." For example, quantization compresses storage by reducing the number of bits in matrix elements, sparsity reduces effective storage by zeroing out less important elements, and pruning directly removes some model parameters or cache entries. Methods like CacheGen are specifically optimized for long-context inference scenarios, attempting to alleviate network latency issues under long sequences.
[0004] Although existing technologies have made some progress in reducing memory footprint, there are still significant limitations that are difficult to overcome, which severely restrict their effectiveness in practical deployments.
[0005] First, the issues of performance loss and accuracy degradation are prominent. Mainstream methods such as quantization and pruning often employ a uniform processing strategy when performing global compression, ignoring the fundamental differences in semantic representation and information importance between the key matrix K and the value matrix V. This leads not only to the loss of key local details during compression but also to a cascading decline in inference quality—specifically manifested as logical breaks in the generated sequence, increased end-to-end latency, or loss of core semantic information, making it difficult to achieve the practical goal of less than 1% performance degradation in inference. This trade-off of "sacrificing accuracy for compression" makes existing technologies unsuitable for scenarios with high accuracy requirements (such as professional document generation and high-precision translation).
[0006] Secondly, computational overhead remains high. Many optimization schemes, in pursuit of higher compression rates, introduce complex auxiliary computational processes: for example, attention-based sparsity requires additional calculation of attention weights between tokens to determine element importance, and multi-head processing necessitates processing and then merging the KV caches of different attention heads separately. These operations significantly increase computational complexity. Even if some methods reduce storage overhead, the additional computational cost prevents an effective improvement in inference speed, and in long-context scenarios, the accumulated computation further prolongs inference latency, contradicting the core purpose of "storage optimization."
[0007] Third, the adaptability to different scenarios is severely lacking. Most existing technologies adopt fixed compression strategies and lack dynamic adjustment mechanisms, making it impossible to flexibly adapt to different task types (such as translation, summarization, and dialogue), sequence lengths (fast response for short sequences, deep inference for long sequences), or hardware environments (high-performance GPUs, edge device CPUs). For example, traditional low-rank methods only focus on global redundancy compression, completely ignoring the need to preserve details in local high-information-density regions, resulting in poor performance in text processing with dense technical terms and tight logical connections; while some local optimization methods are difficult to adapt to the global redundancy control of long sequences, severely limiting their applicable scenarios.
[0008] Fourth, the challenge of balancing global and local factors remains unresolved. Optimizing large-model key-value caching essentially requires balancing two major goals: "global redundancy compression" and "preservation of local details." However, existing technologies lack effective coordination mechanisms: either excessive global compression leads to the loss of local information, or excessive local compensation increases the computational and storage burden, creating a trade-off between "compression ratio and inference performance being mutually exclusive." Furthermore, many methods suffer from insufficient tool support, making it difficult to achieve deep compatibility with mainstream deep learning frameworks (such as PyTorch and TensorFlow) and hardware platforms, further limiting their industrial application.
[0009] It is evident that existing methods cannot simultaneously meet the core requirements of "high compression rate, high fidelity, low computational overhead, and strong adaptability," and a new optimization approach is urgently needed to overcome the storage bottleneck of KV caching. Summary of the Invention
[0010] The present invention aims to at least partially solve one of the technical problems in the aforementioned related technologies.
[0011] Therefore, the purpose of this invention is to provide a large model KV cache compression method, system, product and device based on global low-rank approximation and local residual compensation, which can achieve a balance between efficient compression of global redundancy and accurate preservation of local details, and provide a feasible solution for the deployment of large models in scenarios such as long sequence inference and resource-constrained devices.
[0012] To solve the above-mentioned technical problems, the present invention is implemented as follows: This invention provides a method for compressing large model key-value (KV) caches, the method comprising: S1. Global Low-Rank Approximation: Captures the overall redundancy of the KV matrix and reduces global storage overhead through low-rank decomposition; S2. Local residual compensation: compensates for the loss of local details caused by global low-rank approximation, and accurately preserves the details of high information density areas; S3. Global-Local Complementary Compensation: Integrating the redundancy compression of global low-rank approximation with the detail preservation of local residual compensation to form a complementary and optimized KV cache; S4. Dynamic Complementary Management: Dynamically adjust parameters to optimize performance based on different tasks, sequence lengths, or hardware environments.
[0013] In addition, the large-model KV cache compression method according to the present invention may also have the following additional technical features: In some implementations, the local residual compensation in step S2 introduces multi-scale window analysis and adaptive sparse threshold optimization to dynamically capture local residuals at different levels and retain only significant compensation vectors, thereby reducing noise interference and computational overhead.
[0014] In some implementations, the multi-scale window analysis captures both short-range token associations and long-range contexts simultaneously through fine-grained, medium-grained, and coarse-grained windows to avoid missing information from a single window.
[0015] In some of these embodiments, the compensation coefficient α of the local residual compensation l Based on dynamic adjustment of residual relative intensity, the sparse threshold τ s Real-time optimization based on residual distribution is used to avoid overcompensation or undercompensation of fixed parameters.
[0016] In some implementations, the global-local complementary compensation algorithm in step S3 constructs a closed-loop mechanism of global low-rank approximation-local residual compensation-cooperative reorganization. The global decomposition is divided into local compensation to reduce the total amount of residuals that need to be compensated; local compensation fills in the gaps in the global decomposition to repair the details lost in the global compression, and the two form a positive cycle. By summing multiple overlapping windows at multiple scales, the problem of boundary information breakage in splicing and reassembling is solved.
[0017] In some implementations, the dynamic complementary management in step S4 establishes a dynamic adaptation logic for performance feedback and parameter adjustment, and flexibly switches between compression-first and precision-first modes according to scenario requirements.
[0018] In some implementations, the parameter linkage methods of dynamic complementary management include: linkage based on the triangular relationship of storage-precision-latency, and adjustment and optimization including rank. r ,window w l Threshold τ s Parameters including those included.
[0019] This invention also provides a large model KV cache compression system based on global low-rank approximation and local residual compensation, the system comprising: Global low-rank approximation module: configured to capture the overall redundancy of the KV matrix and reduce global storage overhead through low-rank decomposition; Local residual compensation module: It is configured to compensate for the loss of local details caused by global low-rank approximation and accurately preserve the details of high information density areas; Global-local complementary compensation module: configured to integrate the redundancy compression of global low-rank approximation with the detail preservation of local residual compensation to form a complementary and optimized KV cache; Dynamic Complementary Management Module: Configured to dynamically adjust parameters to optimize performance based on different tasks, sequence lengths, or hardware environments.
[0020] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the large model KV cache compression method based on global low-rank approximation and local residual compensation as described in any of the preceding embodiments.
[0021] This invention also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the large model KV cache compression method based on global low-rank approximation and local residual compensation as described in any of the preceding embodiments.
[0022] Compared with the prior art, the present invention has at least the following beneficial effects: In this embodiment of the invention, the large-model KV cache compression method based on global low-rank approximation and local residual compensation, through the complementary mechanism of global low-rank decomposition and local residual compensation, can significantly reduce the storage requirements of the KV cache while ensuring inference accuracy. This method achieves a balance between high compression ratio and high fidelity, avoids the loss of key information during compression as in traditional methods, and ensures adaptability to different scenarios through a dynamic adjustment mechanism. This innovative technology solves the memory bottleneck problem of large-scale models during inference, and is particularly suitable for deployment in long-sequence inference and resource-constrained devices.
[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0024] Figure 1 This is a flowchart of a large model KV cache compression method based on global low-rank approximation and local residual compensation disclosed in an embodiment of the present invention; Figure 2 This is a flowchart of a local residual compensation algorithm disclosed in one embodiment of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples and application scenarios.
[0027] In some embodiments of the present invention, a large model KV cache compression method based on global low-rank approximation and local residual compensation is provided, such as... Figure 1 As shown, efficient compression and high-performance inference are achieved through four technical steps. The core innovation is a global-local complementary compensation algorithm, which captures redundancy through global low-rank decomposition and complements it with local residual compensation, optimizing the storage and computation efficiency of KV cache. For the local complementary part, this invention introduces an innovative design: a multi-scale adaptive sparse compensation mechanism, which improves the accuracy and efficiency of compensation through hierarchical window analysis and sparse threshold optimization.
[0028] The names and functions of the four modules are as follows: Global low-rank approximation module: Performs low-rank decomposition on the entire KV matrix to compress global redundancy.
[0029] Local residual compensation module: Calculates residual vectors for local regions and performs multi-scale adaptive sparse compensation to preserve details.
[0030] Global-Local Complementary Compensation Algorithm: KV cache optimization is achieved through the coordinated reorganization of global decomposition and local compensation.
[0031] Dynamic complementary management module: dynamically adjusts global rank and local compensation parameters according to task requirements to adapt to different scenarios.
[0032] In some embodiments of the present invention, the purpose of global low-rank approximation is to capture the global redundancy of the KV matrix and compress the overall storage through low-rank decomposition. The input to this module is the key matrix. Value matrix The method involves applying low-rank decomposition to K and V: ; In the formula: Left singular vector matrix, capturing sequence-level patterns; : Diagonal singular value matrix, representing the intensity of the principal components; : Right singular vector matrix, mapped to the original dimension; The rank after dimensionality reduction controls the compression intensity.
[0033] The global low-rank approximation module retains the first r singular values through singular value decomposition (SVD), compresses global redundancy, and reduces storage from O(Nd) to O(Nr+rd).
[0034] In some embodiments of the present invention, the purpose of local residual calculation is to calculate the local residual between the global approximation and the original KV, introducing a multi-scale window to capture details at different levels. Local residual calculation targets multiple window scales. (e.g., w_1=16, w_2=64, w_3=256) Calculate the residuals: ; In the formula: : No. i The token and its first l Submatrices of each scale window; : A subset of global U_k and U_v in a multi-scale window; : Multiscale local residual submatrix.
[0035] The role of local residual calculation is to innovatively capture short-range (fine-grained) and long-range (coarse-grained) details in multi-scale design, avoid the limitations of a single window, and improve the robustness of compensation.
[0036] In some embodiments of the present invention, the purpose of local residual compensation is to enhance local details through adaptive sparse compensation, complementing global compression, and to introduce a sparsity threshold to optimize compensation efficiency. The method involves applying sparse compensation to multi-scale residuals: , ; In the formula: : Compensated multi-scale local submatrix; : No. l The compensation coefficients at each scale are adaptively calculated (e.g., , (for scale weights) : Sparse function, which includes residuals with norms below a threshold Set the elements to zero: S(\Delta)_{j,k}=\begin{cases}\Delta_{j,k}&\|\Delta_{j,k}\|_2\geq\tau_s\\0&\|\Delta_{j,k}\|_2<\tau_s\end{cases}, Adaptation is based on residual distribution (such as median norm).
[0037] The role of local residual compensation: Innovative adaptive sparse compensation retains only significant residuals, reducing noise interference and improving compensation efficiency; multi-scale fusion achieves this through weighted summation (e.g., , To integrate weights and achieve hierarchical complementarity.
[0038] The local residual compensation algorithm is the core module of this invention, aiming to enhance local detail preservation through an adaptive sparse compensation mechanism, complementing the global low-rank approximation. The algorithm's innovation lies in introducing multi-scale window analysis and adaptive sparse threshold optimization, allowing it to dynamically capture local residuals at different levels and retain only significant compensation vectors, thereby reducing noise interference, lowering computational overhead, and improving the accuracy and efficiency of compensation. This design avoids the over-computation of traditional compensation methods, ensuring that model inference quality is maintained under high compression ratios. The algorithm input is the residual submatrix after global low-rank approximation, and the output is the compensated local KV submatrix. The overall complexity is O(log n). ,in Let $L$ be the number of scales (usually $L=3$), $w$ be the maximum window size, and $d$ be the dimension. It can be further optimized to sublinearity through sparsification.
[0039] enter: Multi-scale local residual submatrix: (For each scale) Window size (e.g., 16, 64, 256) Global low-rank approximate submatrix: ; Adaptive parameters: Scale weights Initially This can be optimized through task feedback; initial sparsity threshold (e.g., the 10th percentile of the residual norm).
[0040] Output: Compensated local KV submatrix: (Multi-scale); The fused compensation vector: (Each Token).
[0041] like Figure 2 As shown, the steps of the local residual compensation algorithm include: Step 1. Calculate the adaptive compensation coefficient : Objective: To adaptively adjust the compensation range based on the residual intensity to ensure that high residual regions receive more enhancement; formula: , For scale weights.
[0042] This step, Preset scale weights (e.g., small scale) Emphasizing fine-grained details, large scale (Emphasis on context). Frobenius norm The proportion of the quantized residual relative to the global approximation is adjusted to avoid overcompensation. Innovation: Adaptability enables... Dynamically respond to the information density of different sequence regions.
[0043] Step 2. Adaptive sparse threshold optimization: Objective: To dynamically determine the sparsity threshold Filter out noise residuals and retain only significant details.
[0044] Formula: First, calculate the L2 norm vector of the residual elements. ; in The adjustment factor is 0.5 (empirical value, which can be optimized using the validation set).
[0045] This step has an initial threshold. The median and standard deviation (std) are derived from global residual statistics to ensure that the threshold adapts to the current window distribution. Innovation: Unlike traditional sparsification with a fixed threshold, this adaptive mechanism adjusts in real time according to the residual distribution, improving the targeting and robustness of the compensation.
[0046] Step 3. Apply sparse functions : Objective: To sparsify the residual matrix and reduce computational and storage redundancy. Formula:
[0047] Similarly applied .
[0048] This step checks the norm element by element, setting low-norm elements to zero to achieve sparse representation (achieving a sparsity rate of 20%-50%). Innovation: Combining multi-scale methods, sparse compensation focuses on significant residuals, avoiding generalization errors from global compensation and improving efficiency.
[0049] Step 4. Compensation Calculation: Objective: To add weighted sparse residuals to the global approximation to form a compensation submatrix. Formula: Similarly .
[0050] This step ensures that compensation only enhances the necessary details, thus complementing the overall picture.
[0051] Step 5. Multi-scale fusion: Objective: To integrate multi-scale compensation results to form the final token vector.
[0052] formula: in To normalize the weights.
[0053] This step handles window overlap by weighted summation and fusion of different scales (using moving averages to avoid boundaries). Innovation: Multi-scale fusion is applied for the first time to KV residual compensation, achieving hierarchical complementarity and improving adaptability to long sequences.
[0054] In some embodiments of the present invention, the purpose of global-local reorganization is to integrate global approximation and local compensation to generate a complementary key-value cache. The method is as follows: ; This formula uses multi-scale overlapping windows for summation to avoid boundary effects. The role of global-local reorganization is to form a seamless, complementary key-value cache.
[0055] For the KV cache, calculate its attention and replace KV with K'V'.
[0056] Attention is calculated as follows: .
[0057] In some embodiments of the present invention, the purpose of the dynamic complementary management module is to balance global compression and local compensation according to the task. Method: Adjusting rank Scale set sparse threshold and weight Optimize through performance feedback.
[0058] In this invention, the local residual compensation algorithm is the core module of the global-local complementary compensation algorithm. It aims to enhance the preservation of local details through an adaptive sparse compensation mechanism, complementing the global low-rank approximation. The algorithm's innovation lies in introducing multi-scale window analysis and adaptive sparse threshold optimization, allowing it to dynamically capture local residuals at different levels and retain only significant compensation vectors. This reduces noise interference, lowers computational overhead, and improves the accuracy and efficiency of compensation. This design avoids the over-computation of traditional compensation methods, ensuring that model inference quality is maintained even at high compression ratios.
[0059] The algorithm takes the residual submatrix after global low-rank approximation as input and outputs the compensated local KV submatrix. The overall complexity is O(n log n). ,in Let $L$ be the number of scales (usually $L=3$), $w$ be the maximum window size, and $d$ be the dimension. It can be further optimized to sublinearity through sparsification.
[0060] Example 1: This specific implementation on the LLaMA-7B model was tested using the PyTorch framework on an NVIDIA A100 GPU for a WMT machine translation task with a sequence length of 4096 tokens. Redundancy was compressed through a global low-rank approximation (r=128), and details were preserved by combining multi-scale windows (16, 64, 256) and adaptive sparsity compensation (threshold based on residual norm), resulting in improved inference speed and support for longer sequences.
[0061] Any part of this invention not described in detail can be referred to in the prior art or in the art known to those skilled in the art. This embodiment does not limit such part and will not describe it in detail here.
[0062] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.
Claims
1. A method for compressing large-scale key-value (KV) caches, characterized in that, The method includes: S1. Global Low-Rank Approximation: Captures the overall redundancy of the KV matrix and reduces global storage overhead through low-rank decomposition; S2. Local residual compensation: compensates for the loss of local details caused by global low-rank approximation, and accurately preserves the details of high information density areas; S3. Global-Local Complementary Compensation: Integrating the redundancy compression of global low-rank approximation with the detail preservation of local residual compensation to form a complementary and optimized KV cache; S4. Dynamic Complementary Management: Dynamically adjust parameters to optimize performance based on different tasks, sequence lengths, or hardware environments.
2. The large model KV cache compression method according to claim 1, characterized in that, The local residual compensation in step S2 introduces multi-scale window analysis and adaptive sparse threshold optimization to dynamically capture local residuals at different levels and retain only significant compensation vectors, thereby reducing noise interference and computational overhead.
3. The large model KV cache compression method according to claim 2, characterized in that, The multi-scale window analysis captures both short-range token associations and long-range contexts simultaneously through fine-grained, medium-grained, and coarse-grained windows to avoid missing information from a single window.
4. The large model KV cache compression method according to claim 2, characterized in that, The compensation coefficient α of the local residual compensation l Based on dynamic adjustment of residual relative intensity, the sparse threshold τ s Real-time optimization based on residual distribution is used to avoid overcompensation or undercompensation of fixed parameters.
5. The large model KV cache compression method according to claim 1, characterized in that, In step S3, the global-local complementary compensation algorithm constructs a closed-loop mechanism of global low-rank approximation, local residual compensation, and collaborative reorganization. The global decomposition is divided into local compensation to reduce the total amount of residuals that need to be compensated; local compensation fills in the gaps in the global decomposition to repair the details lost in the global compression, and the two form a positive cycle. By summing multiple overlapping windows at multiple scales, the problem of boundary information breakage in splicing and reassembling is solved.
6. The large model KV cache compression method according to claim 1, characterized in that, Step S4 involves dynamic complementary management, establishing dynamic adaptation logic for performance feedback and parameter adjustment, and flexibly switching between compression-first and precision-first modes according to scenario requirements.
7. The large model KV cache compression method according to claim 6, characterized in that, The parameter linkage methods of dynamic complementary management include: linkage based on the triangular relationship of storage-precision-latency, and adjustment and optimization including rank. r ,window w l Threshold τ s Parameters including those included.
8. A large-model KV cache compression system, characterized in that, The system includes: Global low-rank approximation module: configured to capture the overall redundancy of the KV matrix and reduce global storage overhead through low-rank decomposition; Local residual compensation module: It is configured to compensate for the loss of local details caused by global low-rank approximation and accurately preserve the details of high information density areas; Global-local complementary compensation module: configured to integrate the redundancy compression of global low-rank approximation with the detail preservation of local residual compensation to form a complementary and optimized KV cache; Dynamic Complementary Management Module: Configured to dynamically adjust parameters to optimize performance based on different tasks, sequence lengths, or hardware environments.
9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the large model KV cache compression method according to any one of claims 1-7.
10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the large model KV cache compression method according to any one of claims 1-7.