Kv cache optimization system and method for reducing memory overhead of large language model long text reasoning

Abstract

The application discloses a KV cache optimization system for reducing GPU memory consumption of long text reasoning of a large language model, which comprises an offline optimization module, a pre-filling decoupling module, a cache module and a dynamic reconstruction module: the offline optimization module determines optimal compression parameters of each layer of the model and performs offline decomposition on static weights; the pre-filling decoupling module adaptively decouples input word embeddings into sparse components and residual components and generates structured codes; the cache module is responsible for storing the structured codes to replace traditional KV matrices; and the dynamic reconstruction module uses a custom operator to reconstruct the codes into transient KV tensors in real time during the reasoning stage and releases the codes immediately after attention calculation is completed. The application retains key feature information through an adaptive decoupling mechanism, reduces storage requirements of residual components through a structured coding and low-rank projection mechanism, and efficiently reconstructs compressed sparse codes and low-rank codes through a custom fusion operator, thereby establishing an on-demand calculation and immediate release mechanism of transient key-value tensors.

Description

Technical Field

[0001] This invention relates to the field of long text inference optimization technology in the field of natural language processing, and particularly to a KV cache optimization system and method for reducing the memory overhead of long text inference in large language models. Background Technology

[0002] Currently, Large Language Models (LLMs) have demonstrated powerful capabilities in Natural Language Processing (NLP) and broader Artificial Intelligence (AI) applications, such as content generation and multi-turn dialogue. During inference, LLMs typically execute an autoregressive process, primarily consisting of a prefilling stage for context encoding and a decoding stage for generating individual lexical units. To improve efficiency, modern LLM systems widely employ key-value (KV) caching mechanisms. By storing and reusing computed key and value tensors, redundant computations are avoided, significantly accelerating the generation process.

[0003] Despite the promising application of key-value (KV) caching mechanisms, the inference efficiency of large language models still faces significant challenges. This is primarily because the memory usage of KV caches increases linearly with sequence length. For example, when processing long contextual inputs (such as 128K words), the cache size can easily balloon to hundreds of gigabytes (GB), far exceeding the memory capacity of a single GPU. Therefore, employing effective KV cache compression strategies has become an urgent need for achieving efficient inference with large language models.

[0004] To alleviate this memory bottleneck, researchers have proposed a series of methods based on architecture optimization and system offloading. Some methods focus on improving the model's attention architecture, such as utilizing Multi-Query Attention (MQA) or Grouped Query Attention (GQA) to fundamentally reduce the storage requirements of the key-value cache. Other methods employ offloading strategies, migrating inactive tensors to CPU memory or NVMe storage. For example, some works have designed tiered storage systems to offload infrequently used key-value data to slower storage media. While these methods alleviate memory pressure to some extent, architectural modifications often require costly retraining, and offloading strategies are limited by PCIe bandwidth, which can easily lead to a significant increase in inference latency.

[0005] Given that not all terms in long sequences are equally important for the current generation, terminology-based pruning methods are widely used in key-value (KV) cache compression. These methods advocate dynamically discarding unnecessary terms during inference to maintain a fixed cache budget. For example, SnapKV and PyramidKV identify and retain key terms by analyzing attention scores, discarding historical information with little impact on subsequent generation. While this approach significantly reduces storage overhead, aggressive pruning strategies can lead to the loss of crucial contextual information when handling complex inference tasks requiring long-range dependencies, thus degrading model performance.

[0006] In recent years, quantization and low-rank decomposition methods for dynamic key-value (KV) matrices have received increasing attention. Studies have shown that KV matrices exhibit significant structural redundancy and robustness to quantization. Based on this, some methods (such as KIVI and Quest) introduce low-bit quantization techniques to compress FP16-precision KV matrices to INT4 or even lower precision. Other methods utilize low-rank decomposition techniques to project dynamically generated KV matrices into a low-dimensional subspace for storage. These methods attempt to optimize storage efficiency by reducing numerical precision or feature dimensionality without discarding tokens.

[0007] However, existing low-rank or quantized methods primarily operate on dynamically generated key-value matrices, often neglecting the inherent sparsity of input token embeddings and the low-rank structure of model static weights. This results in a sharp increase in memory overhead in long-context scenarios, becoming a major bottleneck restricting inference efficiency. Summary of the Invention

[0008] This invention provides a KV cache optimization system and method for reducing the memory overhead of long text inference in large language models in order to solve the technical problems existing in the prior art.

[0009] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows: A key-value caching optimization system for reducing the GPU memory overhead of long text inference in large language models includes an offline optimization module, a pre-filling decoupling module, a caching module, and a dynamic reconstruction module. The offline optimization module is used to determine the optimal compression parameters for each layer of the large language model and to perform offline decomposition of the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, and obtains a shared sparse budget, independent key rank values ​​and value rank values. The static key weight matrix and static value weight matrix of the attention mechanism of each layer of the large language model are respectively decomposed using the singular value decomposition submodule, and the corresponding key low-rank basis matrix and value low-rank basis matrix are obtained. The pre-filled decoupling module is used to adaptively decouple the input lexical embeddings into sparse components and residual components based on a shared sparse budget, and generate structured coded data from the sparse components and residual components. The cache module is used to store the structured coded data generated by the pre-filled decoupling module, which replaces the KV matrix. The dynamic reconstruction module is used to reconstruct structured encoded data into transient KV tensors in real time during the inference phase using custom operators, and releases them immediately after the attention calculation is completed.

[0010] Furthermore, the pre-filled decoupling module includes an adaptive decoupling unit, a common sparse component masking and quantization processing unit, a common residual component projection and quantization processing unit, a common structured coding unit, a compression unit, and an independent structured coding unit. The adaptive decoupling unit is used to adaptively decouple the input lexical embeddings into common sparse components and common residual components based on a shared sparse budget. The common sparse component masking and quantization processing unit is used to perform double-layer bit masking and quantization processing on the common sparse components respectively, and obtain the common bit mask matrix and common sparse quantization value accordingly. The common residual component projection and quantization processing unit is used to project and quantize the common residual component sequentially using the bond low-rank basis matrix and the value low-rank basis matrix, respectively, to obtain the quantized value of the bond residual and the quantized value of the value residual. The common structured coding unit is used to perform common structured coding on the common bitmask matrix and common sparse quantization values ​​to generate common structured coded data. The compression unit is used to compress common structured coded data; Independent structured coding units are used to perform independent structured coding on the quantized values ​​of key residuals and value residuals to generate independent structured coded data.

[0011] Furthermore, the dynamic reconstruction module includes a custom fusion operator component and a projection component; the custom fusion operator component includes a common data decompression operator, a common data inverse quantization operator, an independent data inverse quantization operator, a transient key tensor operator, and a transient value tensor operator; the projection component includes a common sparse component projection unit, a key intermediate state matrix projection unit, and a value intermediate state matrix projection unit. The public data decompression operator is used to perform decompression processing on compressed public structured coded data; The public data dequantization operator is used to perform dequantization operations on public structured coded data to obtain the reconstructed public sparse components; The independent data dequantization operator is used to perform dequantization operations on independent structured coded data to obtain the reconstructed key intermediate matrix and value intermediate matrix; The common sparse component projection unit is used to project the common sparse component using the static bond weight matrix and the static value weight matrix respectively, and obtain the bond sparse contribution and the value sparse contribution accordingly. The bond intermediate state matrix projection unit is used to project the bond intermediate state matrix using the low-rank basis matrix of the bond to obtain the bond residual contribution. The key intermediate state matrix projection unit is used to project the value intermediate state matrix using a value low-rank basis matrix to obtain the value residual contribution. The transient bond tensor operator is used to add the bond sparse contribution and the bond residual contribution in real time to obtain the reconstructed transient bond tensor; The transient value tensor operator is used to add the sparse contribution of the value to the residual contribution of the value in real time to obtain the reconstructed transient value tensor.

[0012] This invention also provides a key-value caching optimization method to reduce the memory overhead of long text inference in large language models, the method comprising the following steps: Step 1: Construct an offline optimization module to determine the optimal compression parameters for each layer of the large language model and perform offline decomposition on the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, obtaining a shared sparse budget, independent key-rank values, and value-rank values. The static key-weight matrix and static value-weight matrix of the attention mechanism of each layer of the large language model are decomposed using the singular value decomposition submodule, respectively, to obtain the corresponding low-rank key basis matrix and low-rank value basis matrix. Step 2: Construct a pre-filled decoupling module. Based on a shared sparse budget, the pre-filled decoupling module adaptively decouples the input lexical embeddings into common sparse components and common residual components. Step 3: Perform double-layer bitmasking and quantization on the common sparse components to obtain the common bitmask matrix and common sparse quantization value; use the bond low-rank basis matrix and the value low-rank basis matrix to project and quantize the common residual components in sequence to obtain the bond residual quantization value and the value residual quantization value. Step 4: Use common structured encoding for the common bitmask matrix and common sparse quantization values, and use independent structured encoding for the key residual quantization values ​​and value residual quantization values. Construct a video memory cache module so that the common structured encoded data and independent structured encoded data replace the KV matrix and are stored in the video memory cache module. Step 5: Construct a dynamic reconstruction module. In the dynamic reconstruction module, set up a custom fusion operator component. In each decoding step of the large language model inference stage, call the custom fusion operator component to perform decompression and dequantization operations on the common structured encoded data to obtain the reconstructed common sparse components. Then, use the static key weight matrix and the static value weight matrix to project the common sparse components respectively to obtain the key sparse contribution and the value sparse contribution. Step 6: Perform dequantization operation on the independent structured coded data using a custom fusion operator component to obtain the reconstructed key intermediate state matrix and value intermediate state matrix; and project the reconstructed key and value intermediate state matrices using the key and value low-rank basis matrices to obtain the corresponding key residual contribution and value residual contribution. Step 7: During the online long text inference process of the large language model, the key sparse contribution and key residual contribution are added in real time within the custom fusion operator component to obtain the reconstructed transient key tensor, and the value sparse contribution and value residual contribution are added to obtain the reconstructed transient value tensor. The transient key tensor and transient value tensor are released immediately after the attention calculation is completed.

[0013] Furthermore, corresponding to the large language model, the first Layer, assuming: shared sparse budget is The key rank value is The rank value is The compression configuration parameter recipe is as follows , The static key weight matrix is: The static value weight matrix is Step 1 includes the following sub-steps: Step 1-1, let: In the evolutionary search submodule, The population size is The number of iterations is The rate of variation Crossover rate ; The The formula is ;Include The population set of the candidate formulations is , The current globally optimal formula is ; Assignment , , , ,initialization and ; Step 1-2, Perform differential mutation operation: In each generation iteration, traverse the current population. For the first One recipe From the population Three different recipes are randomly selected. The following differential mutation formula is used to generate candidate formulations: ; In the formula: for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for Candidate formulations for mutations; Steps 1-3, perform crossover operation: for the first... One recipe With probability Select from variant candidate formulations The value in the middle, thus generating the offspring recipe, let For the corresponding number One recipe Offspring formula; Steps 1-4, apply projection constraints: adjust the offspring recipe The parameter values ​​are projected to a preset range of valid values ​​to ensure a shared sparse budget. And rank value If the integer is positive, a feasible offspring formula is obtained; Steps 1-5, Fitness Evaluation: Apply feasible progeny formulas to the large language model, perform inference tests on the validation set, calculate validation loss or accuracy, and use the validation loss or accuracy as the fitness score. Steps 1-6, Selection and Update: This involves generating a set of recipes for all offspring. with parental population Merge, sort by fitness score and retain the top score. The optimal recipe is used as the next generation population, and the globally optimal recipe is updated simultaneously. ; Steps 1-7, for the static key weight matrix Perform key singular value decomposition; the decomposition formula is as follows: ; In the formula: Static key weight matrix The left singular vector matrix; Static key weight matrix The right singular vector matrix; For containing static key weight matrix A diagonal matrix of singular values; Sort the key singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The _n_ key singular values ​​and their corresponding key singular vectors are used to construct the low-rank basis matrix of the keys according to the following formula: ; ; In the formula: To sort by first The left key singular vector matrix corresponding to each key singular value; To sort by first The right-key singular vector matrix corresponding to each key singular value; For including the previous A diagonal matrix of singular values ​​of each key; The left-hand low-rank basis matrix; ; The right-click low-rank basis matrix; ; For the hidden layer dimension; For projection dimensions; Steps 1-8: Adjust the static value weight matrix. The singular value decomposition of the execution value is performed, and the decomposition formula is as follows: ; In the formula: Static value weight matrix The left singular vector matrix; Static value weight matrix The right singular vector matrix; Static value weight matrix A diagonal matrix of singular values; Sort the singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The singular values ​​and their corresponding singular vectors are used to construct a low-rank basis matrix of values ​​according to the following formula: ; ; In the formula: To sort by first The left-value singular vector matrix corresponding to the singular values ​​of each singular value; To sort by first The rvalue singular vector matrix corresponding to the singular values ​​of each singular value; For including the previous A diagonal matrix with singular values; It is an lvalued low-rank basis matrix; ; It is an rvalued low-rank basis matrix; .

[0014] Furthermore, step 2 includes the following sub-steps: Step 2-1, corresponding to the first step of the large language model. Layer, let: input lexical embedding be , The amplitude threshold is Shared sparse budget is ; Based on shared sparse budget Calculate the input word embedding amplitude threshold This ensures that the proportion of elements in the input lexical embedding whose absolute value is less than the threshold conforms to the sparse budget. Step 2-2: Extract common sparse components and use amplitude thresholding. Embedding of input tokens Each element in the dataset is filtered to obtain the common sparse component. The formula for calculating the common sparse component is as follows: ; In the formula: For common sparse components; ; Represents element-level sparsification operator functions; Batch size; For context length; For the hidden layer dimension; The function expression is as follows: ; In the formula: Embedding of input tokens Elements in; Steps 2-3: Extract the common residual component. Subtract the common sparse component from the input token embedding to obtain the common residual component. The formula for calculating the common residual component is: ; In the formula: For common residual components; .

[0015] Furthermore, step 3 includes the following sub-steps: Step 3-1, corresponding to the first step of the large language model. Layer, let: input lexical embedding be , The amplitude threshold is Shared sparse budget is The common sparse component is The key rank value is The rank value is ; For common sparse components Perform bitmasking operations to obtain the outer bitmask, which records the positions of non-zero elements of the common sparse components. The formula for calculating the outer bitmask is as follows: ; In the formula: for The outer bitmask; ; Batch size; For context length; For the hidden layer dimension; For indicator functions; Step 3-2, for common sparse components Perform INT8 uniform quantization to obtain a common sparse quantized value, and then perform a bitmasking operation on the common sparse quantized value to obtain the inner bitmask. The formula for calculating the inner bitmask is as follows: ; ; In the formula: For common sparse quantization values; This indicates that INT8 uniform quantization is being performed; for Inner bitmask; ; Step 3-3: Construct the bond residual code. Project the common residual components onto the low-dimensional subspace using the low-rank basis matrix of the bonds to obtain the bond intermediate state matrix. Then, perform INT4 uniform quantization on the bond intermediate state matrix to obtain the quantized value of the bond residual. The formula for calculating the quantized value of the bond residual is as follows: ; ; In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​the bond matrix; ; For common residual components; The left-hand low-rank basis matrix; This represents the INT4 uniform quantization function; Steps 3-4 involve constructing value residual codes. Common residual components are projected onto a low-dimensional subspace using a low-rank value basis matrix to obtain a value intermediate state matrix. This matrix is ​​then subjected to INT4 uniform quantization to obtain the value residual quantized values. The formula for calculating the value residual quantized values ​​is as follows: ; ; In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​a value. ; It is an left-valued low-rank basis matrix.

[0016] Furthermore, step 4 includes the following sub-steps: Step 4-1, corresponding to the first step of the large language model. The layer allocates a common storage area in the video memory to store the following common structured encoded data: ; where the outer bit mask and inner bitmask Compressed storage is performed using a bit-packed format; Step 4-2: Allocate a separate storage area in the video memory to store the following independent structured coded data: ; Step 4-3: Construct a cache index, establish a hierarchical mapping relationship, and associate the common structured encoded data with the independent structured encoded data to the first level of the large language model. This layer serves as the sole persistent representation of the key-value data within that layer, replacing the original KV matrix storage.

[0017] Furthermore, step 5 includes the following sub-steps: Step 5-1, corresponding to the first step of the large language model. In the first layer, a custom fusion operator component is started. The non-zero memory offset is calculated directly from the common bit mask using the hardware built-in instructions. The common sparse quantization value is read based on the non-zero memory offset and the inverse quantization operation is performed to obtain the reconstructed common sparse components. Step 5-2: In the custom fusion operator component, perform sparse matrix multiplication operations on the static key weight matrix and the static value weight matrix with the reconstructed common sparse components to obtain the key sparse contribution and value sparse contribution, respectively. The calculation formulas are as follows: ; ; In the formula: Static key weight matrix; Static value weight matrix; For the reconstructed common sparse components; Contribute to bond sparsity; This contributes to the sparse value.

[0018] Furthermore, step 6 includes the following sub-steps: Step 6-1, corresponding to the first step of the large language model. In the custom fusion operator component, the key residual quantized value and the value residual quantized value are read and dequantized to obtain the reconstructed key intermediate state matrix and value intermediate state matrix, respectively. Step 6-2: Perform dense matrix multiplication on the low-rank bond basis matrix and the low-rank value basis matrix, corresponding to the reconstructed intermediate bond matrix and the intermediate value matrix, to obtain the bond residual contribution and the value residual contribution, calculated as follows: ; ; In the formula: Contribution to bond residuals; ; Contribution to residual values; ; Batch size; For context length; For projection dimensions; This is the reconstructed bond intermediate state matrix; This is the reconstructed intermediate state matrix; The right-click low-rank basis matrix; ; It is an rvalued low-rank basis matrix; ; The key rank value; The value is the rank value.

[0019] The advantages and positive effects of this invention are as follows: This invention utilizes the inherent sparsity of input lexical embeddings through an adaptive decoupling mechanism, effectively preserving key feature information; it leverages the low-rank structure of the model's static weights through structured encoding and low-rank projection mechanisms, significantly reducing the storage requirements of residual components; it provides hierarchically adaptive optimal compression parameter configuration using an evolutionary search strategy; and it efficiently reconstructs the compressed sparse and low-rank encodings through a custom fusion operator component, establishing an on-demand computation and instant release mechanism for transient key-value tensors. This achieves a balance between reducing GPU memory usage and maintaining inference accuracy, alleviating the efficiency bottleneck caused by GPU memory constraints in long text inference in existing technologies. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the working principle of a KV cache optimization system for reducing the memory overhead of long text inference in large language models according to the present invention.

[0021] Figure 2 This is a flowchart of a KV cache optimization method for reducing the memory overhead of long text inference in large language models according to the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] Please see Figures 1 to 2 A key-value caching optimization system for reducing the GPU memory overhead of long text inference in large language models is proposed. This system includes an offline optimization module, a pre-filling decoupling module, a caching module, and a dynamic reconstruction module. The offline optimization module is used to determine the optimal compression parameters for each layer of the large language model and to perform offline decomposition of the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, and obtains a shared sparse budget, independent key rank values ​​and value rank values. The static key weight matrix and static value weight matrix of the attention mechanism of each layer of the large language model are respectively decomposed using the singular value decomposition submodule, and the corresponding key low-rank basis matrix and value low-rank basis matrix are obtained. The pre-filled decoupling module is used to adaptively decouple the input lexical embeddings into sparse components and residual components based on a shared sparse budget, and generate structured coded data from the sparse components and residual components. The cache module is used to store the structured coded data generated by the pre-filled decoupling module, which replaces the KV matrix. The dynamic reconstruction module is used to reconstruct structured encoded data into transient KV tensors in real time during the inference phase using custom operators, and releases them immediately after the attention calculation is completed.

[0024] Preferably, the pre-filled decoupling module may include an adaptive decoupling unit, a common sparse component masking and quantization processing unit, a common residual component projection and quantization processing unit, a common structured coding unit, a compression unit, and an independent structured coding unit. The adaptive decoupling unit is used to adaptively decouple the input lexical embeddings into common sparse components and common residual components based on a shared sparse budget. The common sparse component masking and quantization processing unit is used to perform double-layer bit masking and quantization processing on the common sparse components respectively, and obtain the common bit mask matrix and common sparse quantization value accordingly. The common residual component projection and quantization processing unit is used to project and quantize the common residual component sequentially using the bond low-rank basis matrix and the value low-rank basis matrix, respectively, to obtain the quantized value of the bond residual and the quantized value of the value residual. The common structured coding unit is used to perform common structured coding on the common bitmask matrix and common sparse quantization values ​​to generate common structured coded data. The compression unit is used to compress common structured coded data; Independent structured coding units are used to perform independent structured coding on the quantized values ​​of key residuals and value residuals to generate independent structured coded data.

[0025] Preferably, the dynamic reconstruction module may include a custom fusion operator component and a projection component; the custom fusion operator component includes a common data decompression operator, a common data inverse quantization operator, an independent data inverse quantization operator, a transient key tensor operator, and a transient value tensor operator; the projection component includes a common sparse component projection unit, a key intermediate state matrix projection unit, and a value intermediate state matrix projection unit; The public data decompression operator is used to perform decompression processing on compressed public structured coded data; The public data dequantization operator is used to perform dequantization operations on public structured coded data to obtain the reconstructed public sparse components; The independent data dequantization operator is used to perform dequantization operations on independent structured coded data to obtain the reconstructed key intermediate matrix and value intermediate matrix; The common sparse component projection unit is used to project the common sparse component using the static bond weight matrix and the static value weight matrix respectively, and obtain the bond sparse contribution and the value sparse contribution accordingly. The bond intermediate state matrix projection unit is used to project the bond intermediate state matrix using the low-rank basis matrix of the bond to obtain the bond residual contribution. The key intermediate state matrix projection unit is used to project the value intermediate state matrix using a value low-rank basis matrix to obtain the value residual contribution. The transient bond tensor operator is used to add the bond sparse contribution and the bond residual contribution in real time to obtain the reconstructed transient bond tensor; The transient value tensor operator is used to add the sparse contribution of the value to the residual contribution of the value in real time to obtain the reconstructed transient value tensor.

[0026] This invention also provides a key-value caching optimization method to reduce the memory overhead of long text inference in large language models, the method comprising the following steps: Step 1: Construct an offline optimization module to determine the optimal compression parameters for each layer of the large language model and perform offline decomposition on the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, obtaining a shared sparse budget, independent key-rank values, and value-rank values. The static key-weight matrix and static value-weight matrix of the attention mechanism of each layer of the large language model are decomposed using the singular value decomposition submodule, respectively, to obtain the corresponding low-rank key basis matrix and low-rank value basis matrix.

[0027] The key rank value refers to the maximum number of singular key values ​​that are truncated and retained when performing singular value decomposition on the static key weight matrix, as determined by the evolutionary search submodule.

[0028] The value rank refers to the number of maximum singular values ​​that are truncated and retained when performing singular value decomposition on the static value weight matrix, as determined by the evolutionary search submodule.

[0029] The singular value decomposition (SVD) submodule can include a key SVD unit and a value SVD unit. The static key weight matrices of each layer of the large language model are decomposed using the key SVD unit in the SVD submodule to obtain a key low-rank basis matrix. The static value weight matrices of each layer of the large language model are decomposed using the value SVD unit in the SVD submodule to obtain a value low-rank basis matrix.

[0030] Step 2: Construct a pre-filled decoupling module. Based on a shared sparse budget, the pre-filled decoupling module is used to adaptively decouple the input lexical embeddings into common sparse components and common residual components.

[0031] Step 3: Perform double-layer bitmasking and quantization on the common sparse components to obtain the common bitmask matrix and common sparse quantization value; use the bond low-rank basis matrix and the value low-rank basis matrix to project and quantize the common residual components in sequence to obtain the bond residual quantization value and the value residual quantization value.

[0032] The common residual components are projected and quantized sequentially using a low-rank bond basis matrix to obtain the quantized bond residual values. The common residual components are then projected and quantized sequentially using a low-rank value basis matrix to obtain the quantized value value.

[0033] Step 4: Use common structured encoding for the common bitmask matrix and common sparse quantization values, and use independent structured encoding for the key residual quantization values ​​and value residual quantization values. Construct a video memory cache module so that the common structured encoded data and independent structured encoded data replace the KV matrix and are stored in the video memory cache module.

[0034] The common bitmask matrix and common sparse quantization values ​​are constructed using common structured coding to form common structured coded data. The key residual quantization values ​​and value residual quantization values ​​are constructed using independent structured coding to form independent structured coded data. The common structured coded data and independent structured coded data are then stored in the video memory cache module in place of the KV matrix.

[0035] Step 5: Construct a dynamic reconstruction module. In the dynamic reconstruction module, set up a custom fusion operator component. In each decoding step of the large language model inference stage, call the custom fusion operator component to perform decompression and dequantization operations on the common structured encoded data to obtain the reconstructed common sparse components. Then, use the static key weight matrix and the static value weight matrix to project the common sparse components respectively to obtain the key sparse contribution and the value sparse contribution.

[0036] The bond sparse contribution is obtained by projecting the common sparse components onto the static bond weight matrix. The value sparse contribution is obtained by projecting the common sparse components onto the static value weight matrix.

[0037] Step 6: Perform dequantization operation on the independent structured coded data using a custom fusion operator component to obtain the reconstructed key intermediate state matrix and value intermediate state matrix; and project the reconstructed key and value intermediate state matrices using the key and value low-rank basis matrices to obtain the corresponding key residual contribution and value residual contribution.

[0038] By using a custom fusion operator component, inverse quantization is performed on the key residual quantization values ​​in the independent structured coded data to obtain the reconstructed key intermediate state matrix; by using a custom fusion operator component, inverse quantization is performed on the value residual quantization values ​​in the independent structured coded data to obtain the reconstructed value intermediate state matrix.

[0039] The bond residual contribution is obtained by projecting the reconstructed bond intermediate state matrix onto the low-rank bond basis matrix. Similarly, the value residual contribution is obtained by projecting the reconstructed value intermediate state matrix onto the low-rank value basis matrix.

[0040] Step 7: During the online long text inference process of the large language model, the key sparse contribution and key residual contribution are added in real time within the custom fusion operator component to obtain the reconstructed transient key tensor, and the value sparse contribution and value residual contribution are added to obtain the reconstructed transient value tensor. The transient key tensor and transient value tensor are released immediately after the attention calculation is completed.

[0041] In the context of current Large Language Models (LLMs), shared sparse budget refers to how to allocate resources (such as computing power, memory, and inference time) to different sparsification mechanisms (such as MoE's "conditional computation" and Enggram's "conditional memory") under constraints of a fixed total number of parameters and computational overhead (FLOPs) to achieve optimal model performance. The core goal of this sharing mechanism is to improve resource utilization efficiency, reduce costs, and support concurrent services.

[0042] The shared sparse budget is determined by the evolutionary search submodule and is used to compute the input lexical embedding magnitude threshold.

[0043] Structured coding refers to encoding using a specific coding structure.

[0044] Common structured coding refers to the data set composed of the corresponding common bitmask matrix and common sparse quantization value after performing double-layer bitmasking and quantization processing on common sparse components.

[0045] Independent structured coding refers to the data set composed of the corresponding generated key residual quantized values ​​and value residual quantized values ​​after projecting and quantizing the common residual components sequentially using the key low-rank basis matrix and the value low-rank basis matrix respectively.

[0046] The custom fusion operator component is a component that combines basic operators to create custom combinatorial operators and automatically merges these basic operators to improve the performance of custom combinatorial operators.

[0047] The core of the attention calculation module of the large language model is the self-attention mechanism, which is specifically implemented as multi-head self-attention.

[0048] This module is a core component of the Transformer architecture, responsible for calculating the degree of association between each word (or token) in the input sequence and all other words, thereby dynamically generating a representation for each word containing rich contextual information. The attention calculation module implements attention calculation.

[0049] Preferably, the corresponding large language model is the first Layers can be configured with a shared sparse budget. The key rank value is The rank value is The compression configuration parameter recipe is as follows , The static key weight matrix is: The static value weight matrix is Step 1 may include the following sub-steps: Step 1-1, can be set as follows: In the evolutionary search submodule, The population size is The number of iterations is The rate of variation Crossover rate ; The The formula is ;Include The population set of the candidate formulations is , The current globally optimal formula is ; Assignment , , , ,initialization and ; Step 1-2, Perform differential mutation operation: In each generation iteration, traverse the current population. For the first One recipe From the population Three different recipes are randomly selected. The following differential mutation formula is used to generate candidate formulations: ; In the formula: for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for Candidate formulations for mutations; Steps 1-3, perform crossover operation: for the first... One recipe With probability Select from variant candidate formulations The value in the middle, thus generating the offspring recipe, let For the corresponding number One recipe Offspring formula; Steps 1-4, apply projection constraints: adjust the offspring recipe The parameter values ​​are projected to a preset range of valid values ​​to ensure a shared sparse budget. And rank value If the integer is positive, a feasible offspring formula is obtained; Steps 1-5, Fitness Evaluation: Apply feasible progeny formulas to the large language model, perform inference tests on the validation set, calculate validation loss or accuracy, and use the validation loss or accuracy as the fitness score. Steps 1-6, Selection and Update: This involves generating a set of recipes for all offspring. with parental population Merge, sort by fitness score and retain the top score. The optimal recipe is used as the next generation population, and the globally optimal recipe is updated simultaneously. ; Steps 1-7, for the static key weight matrix Perform key singular value decomposition; the decomposition formula is as follows: ; In the formula: Static key weight matrix The left singular vector matrix; Static key weight matrix The right singular vector matrix; For containing static key weight matrix A diagonal matrix of singular values; Sort the key singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The _n_ key singular values ​​and their corresponding key singular vectors are used to construct the low-rank basis matrix of the keys according to the following formula: ; ; In the formula: To sort by first The left key singular vector matrix corresponding to each key singular value; To sort by first The right-key singular vector matrix corresponding to each key singular value; For including the previous A diagonal matrix of singular values ​​of each key; The left-hand low-rank basis matrix; ; The right-click low-rank basis matrix; ; For the hidden layer dimension; For projection dimensions; Steps 1-8: Adjust the static value weight matrix. The singular value decomposition of the execution value is performed, and the decomposition formula is as follows: ; In the formula: Static value weight matrix The left singular vector matrix; Static value weight matrix The right singular vector matrix; Static value weight matrix A diagonal matrix of singular values; Sort the singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The singular values ​​and their corresponding singular vectors are used to construct a low-rank basis matrix of values ​​according to the following formula: ; ; In the formula: To sort by first The left-value singular vector matrix corresponding to the singular values ​​of each singular value; To sort by first The rvalue singular vector matrix corresponding to the singular values ​​of each singular value; For including the previous A diagonal matrix with singular values; It is an lvalued low-rank basis matrix; ; It is an rvalued low-rank basis matrix; .

[0050] A basis matrix is ​​a matrix composed of linearly independent basis vectors in the column space or row space of a matrix.

[0051] In singular value decomposition (SVD) or eigenvalue decomposition of a matrix, the basis matrix typically refers to an orthogonal matrix composed of eigenvectors or singular vectors. For example, in SVD, the matrix (W = U) ), where the column vectors of U and V are orthogonal bases corresponding to the row and column spaces, respectively. W is a g×h matrix, and U is a g×g orthogonal matrix. V is a g×h diagonal matrix, and V is an h×h orthogonal matrix.

[0052] Here, U is called the left singular vector matrix, and its column vectors are called left singular vectors. These vectors are matrices. The eigenvectors form a set of orthonormal bases for the output space (row space).

[0053] In dimensionality reduction, the first few columns of U can be used to project the data into a lower-dimensional space while retaining the main information.

[0054] V is called the right singular vector matrix, and its column vectors are called right singular vectors. These vectors are matrices. The eigenvectors form a set of orthonormal bases for the input space (column space).

[0055] In dimensionality reduction, the first few columns of V can be used to project the data into a lower-dimensional space and extract the main features.

[0056] It is a diagonal matrix, and the non-negative elements on its diagonal are called singular values, which are arranged in descending order and represent the "strength" or "importance" of the matrix in the direction of the corresponding singular vector.

[0057] By selecting the top K largest singular values, dimensionality reduction can be achieved, discarding unimportant information, which can be used for data compression and noise reduction.

[0058] The static key weight matrix singular value decomposes into a left key singular vector matrix and a right key singular vector matrix, and the static key weight matrix singular value decomposes into an left value singular vector matrix and a right value singular vector matrix. Let the key rank be... The rank value is .

[0059] A left-key low-rank basis matrix refers to a matrix whose order is first... The left key singular vector matrix corresponding to each key singular value and containing the previous key singular values. The product of the squares of all elements of a diagonal matrix with singular key values ​​raised to the power of 1 / 2.

[0060] The right-click low-rank basis matrix refers to the matrix that is sorted first. The right key singular vector matrix corresponding to each key singular value and containing the previous key singular values. The product of the squares of all elements of a diagonal matrix with singular key values ​​raised to the power of 1 / 2.

[0061] Lvalued low-rank basis matrices are those that are ordered first. The singular vector matrix corresponding to the singular values ​​of each singular value and containing the first singular value... The product of the squares of all elements of a diagonal matrix with singular values ​​raised to the power of 1 / 2.

[0062] Rvalued low-rank basis matrices are those that are ordered first. The rvalue singular vector matrix corresponding to the singular values ​​of each singular value and containing the first singular value... The product of the squares of all elements of a diagonal matrix with singular values ​​raised to the power of 1 / 2.

[0063] The evolutionary search submodule refers to an intelligent search method inspired by biological evolution mechanisms, widely used in complex optimization problems such as automated machine learning, neural network architecture search, agent design, and suggestion optimization. Its core idea is to automatically explore and approximate the optimal solution by simulating the evolutionary process of "population—mutation—selection—survival of the fittest."

[0064] Core components and processes: Evolutionary search modules typically include the following four key steps: Initial population generation: Randomly generate a set of candidate solutions (such as machine learning pipelines, neural network structures, keyword sets, or combinations of agent modules) to form the initial population.

[0065] Parent selection: Individuals with good fitness (such as model accuracy, search conversion rate, and task completion rate) are selected as parents to reproduce the next generation. Common strategies include: Proportion selection, ranking selection, tournament selection (controlling the balance between development and exploration by setting the size of the candidate pool).

[0066] Crossover and Mutation: Crossover (recombination): Exchanging parts of the genes (such as hyperparameters, modules, keywords) between two parents to generate a new solution.

[0067] Mutation: Randomly perturbing certain parameters of an individual introduces diversity and avoids getting trapped in local optima.

[0068] Survivor selection (population regeneration): Replacing underperforming or older individuals in the population with newly generated offspring to maintain population size. For example: Aging strategy: Remove the oldest individuals, retain the most recent trials, and enhance sensitivity to recent developments.

[0069] Elite retention: Directly retain the best individuals to ensure that performance does not degrade.

[0070] Preferably, step 2 may include the following sub-steps: Step 2-1, corresponding to the first step of the large language model. Layer, which can be configured such that the input lexical embedding is: , The amplitude threshold is Shared sparse budget is ; Based on shared sparse budget Calculate the input word embedding amplitude threshold This ensures that the proportion of elements in the input lexical embedding whose absolute value is less than the threshold conforms to the sparse budget. Step 2-2: Extract common sparse components and use amplitude thresholding. Embedding of input tokens Each element in the dataset is filtered to obtain the common sparse component. The formula for calculating the common sparse component is as follows: ; In the formula: For common sparse components; ; Represents element-level sparsification operator functions; Batch size; For context length; For the hidden layer dimension; The function expression is as follows: ; In the formula: Embedding of input tokens Elements in; Steps 2-3: Extract the common residual component. Subtract the common sparse component from the input token embedding to obtain the common residual component. The formula for calculating the common residual component is: ; In the formula: For common residual components; .

[0071] Preferably, step 3 may include the following sub-steps: Step 3-1, corresponding to the first step of the large language model. Layer, which can be configured such that the input lexical embedding is: , The amplitude threshold is Shared sparse budget is The common sparse component is The key rank value is The rank value is ; For common sparse components Perform bitmasking operations to obtain the outer bitmask, which records the positions of non-zero elements of the common sparse components. The formula for calculating the outer bitmask is as follows: ; In the formula: for The outer bitmask; ; Batch size; For context length; For the hidden layer dimension; For indicator functions; Step 3-2, for common sparse components Perform INT8 uniform quantization to obtain a common sparse quantized value, and then perform a bitmasking operation on the common sparse quantized value to obtain the inner bitmask. The formula for calculating the inner bitmask is as follows: ; ; In the formula: For common sparse quantization values; This indicates that INT8 uniform quantization is being performed; for Inner bitmask; ; Step 3-3: Construct the bond residual code. Project the common residual components onto the low-dimensional subspace using the low-rank basis matrix of the bonds to obtain the bond intermediate state matrix. Then, perform INT4 uniform quantization on the bond intermediate state matrix to obtain the quantized value of the bond residual. The formula for calculating the quantized value of the bond residual is as follows: ; ; In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​the bond matrix; ; For common residual components; The left-hand low-rank basis matrix; This represents the INT4 uniform quantization function; Steps 3-4 involve constructing value residual codes. Common residual components are projected onto a low-dimensional subspace using a low-rank value basis matrix to obtain a value intermediate state matrix. This matrix is ​​then subjected to INT4 uniform quantization to obtain the value residual quantized values. The formula for calculating the value residual quantized values ​​is as follows: ; ; In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​a value. ; It is an left-valued low-rank basis matrix.

[0072] Preferably, step 4 may include the following sub-steps: Step 4-1, corresponding to the first step of the large language model. The layer allocates a common storage area in the video memory to store the following common structured encoded data: ; where the outer bit mask and inner bitmask Compressed storage is performed using a bit-packed format; Step 4-2: Allocate a separate storage area in the video memory to store the following independent structured coded data: ; Step 4-3: Construct a cache index, establish a hierarchical mapping relationship, and associate the common structured encoded data with the independent structured encoded data to the first level of the large language model. This layer serves as the sole persistent representation of the key-value data within that layer, replacing the original KV matrix storage.

[0073] Preferably, step 5 may include the following sub-steps: Step 5-1, corresponding to the first step of the large language model. In the first layer, a custom fusion operator component is started. The non-zero memory offset is calculated directly from the common bit mask using the hardware built-in instructions. The common sparse quantization value is read based on the non-zero memory offset and the inverse quantization operation is performed to obtain the reconstructed common sparse components. Step 5-2: In the custom fusion operator component, perform sparse matrix multiplication operations on the static key weight matrix and the static value weight matrix with the reconstructed common sparse components to obtain the key sparse contribution and value sparse contribution, respectively. The calculation formulas are as follows: ; ; In the formula: Static key weight matrix; Static value weight matrix; For the reconstructed common sparse components; Contribute to bond sparsity; This contributes to the sparse value.

[0074] Preferably, step 6 may include the following sub-steps: Step 6-1, corresponding to the first step of the large language model. In the custom fusion operator component, the key residual quantized value and the value residual quantized value are read and dequantized to obtain the reconstructed key intermediate state matrix and value intermediate state matrix, respectively. Step 6-2: Perform dense matrix multiplication on the low-rank bond basis matrix and the low-rank value basis matrix, corresponding to the reconstructed intermediate bond matrix and the intermediate value matrix, to obtain the bond residual contribution and the value residual contribution, calculated as follows: ; ; In the formula: Contribution to bond residuals; ; Contribution to residual values; ; Batch size; For context length; For projection dimensions; This is the reconstructed bond intermediate state matrix; This is the reconstructed intermediate state matrix; The right-click low-rank basis matrix; ; It is an rvalued low-rank basis matrix; ; The key rank value; The value is the rank value.

[0075] Preferably, step 7 may include the following sub-steps: Step 7-1, corresponding to the first step of the large language model. In the custom fusion operator component, the sparse key contribution obtained in step 5 and the residual key contribution obtained in step 6 are added element-wise to obtain the reconstructed transient key tensor; simultaneously, the sparse value contribution obtained in step 5 and the residual value contribution obtained in step 6 are added element-wise to obtain the reconstructed transient value tensor. The calculation formula is as follows: ; ; In the formula: For the reconstructed transient key tensor; For the reconstructed transient value tensor; Contribute to bond sparsity; To contribute sparsity to values; Contribution to bond residuals; The contribution of the residual.

[0076] Step 7-2, convert the transient key tensor and transient value tensor Input the attention calculation module, perform the dot product operation, and release it immediately after the attention output calculation at the current time step is completed.

[0077] The present invention also provides an apparatus for optimizing the KV cache to reduce the memory overhead of long text inference in large language models, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the KV cache optimization method for reducing the memory overhead of long text inference in large language models as described above.

[0078] The workflow and working principle of the present invention will be further described below with reference to preferred embodiments: Step S1, corresponding to the first step of the large language model. A layer-by-layer offline optimization module is constructed, and an evolutionary search strategy is used to determine the compression configuration parameters of each layer of the model, thereby obtaining the shared sparse budget. and independent key rank values Sum and rank ; the static matrix Perform singular value decomposition on each matrix to obtain the low-rank basis matrix of the bond. Sum-valued low-rank basis matrix ; The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule, which respectively execute the evolutionary search strategy and the singular value decomposition operation.

[0079] The process of determining parameters using an evolutionary search strategy in step S1 specifically includes: Step S1-1, Define the recipe set The recipe set contains a shared sparse budget for each layer of the model. and independent key rank values Sum and rank ,Right now Set population size Number of iterations Variation rate and cross rate Initialization includes Population of candidate formulations And record the current globally optimal recipe. .

[0080] In one embodiment of the present invention, the population Size is 16, number of iterations The variability is 8. and cross rate The values ​​are set to 0.5 and 0.3 respectively. The shared sparse budget... The range is [0.25, 0.95], rank value and The range is [32, 256].

[0081] Step S1-2: Perform differential mutation operation. In each generation iteration, traverse the current population. For the first One recipe Three distinct recipes are randomly selected from the population. Candidate mutations are generated based on the difference mutation formula. : ; Steps S1-3: Perform crossover operations on the first... One recipe With probability Select from variant candidates The value in the middle, thus generating the offspring recipe. .

[0082] Steps S1-4: Perform projection constraints to apply the offspring formula. The parameter values ​​are projected to a preset range of valid values ​​to ensure a shared sparse budget. And rank value If the integer is positive, a feasible offspring formula is obtained.

[0083] Steps S1-5: Fitness assessment, applying feasible progeny formulations to the large language model, and validating them on the validation set. Inference tests are performed on the platform, and the validation loss or accuracy is calculated as the fitness score.

[0084] In one embodiment of the present invention, the verification set It is constructed by randomly sampling clues from the general C4 dataset.

[0085] The General C4 dataset, short for Colossal Clean Crawled Corpus, is a large-scale, high-quality English text dataset extracted by Google from the Common Crawl project and rigorously cleaned and processed. It is widely used to train and evaluate large language models (LLMs), such as Google's T5 model, and is also used as a training data source for models like LLaMA and GPT-3.

[0086] Steps S1-6, Selection and Update, will generate a set of all offspring recipes. with parental population Merge, sort by fitness score and retain the top score. The optimal recipe is used as the next generation population, and the globally optimal recipe is updated simultaneously. .

[0087] Furthermore, the process of performing singular value decomposition on the static weight matrix in step S1 specifically includes: Steps S1-7: Construct the low-rank basis matrix of the bonds, based on the bond rank values ​​determined in step S1. , for the static weight matrix of the layer Perform singular value decomposition, the decomposition form is as follows ; before truncation Construct a low-rank basis matrix from the 1st maximal singular value and its corresponding singular vector. and Its calculation formula is expressed as: ; .

[0088] Steps S1-8: Construct a low-rank basis matrix based on the key rank determined in step S1. , for the static weight matrix of the layer Perform singular value decomposition, and construct a low-rank basis matrix using the same principle as in step S17. and .

[0089] Step S2: Construct a pre-filled decoupling module based on the shared sparse budget determined in step S1. Embed the input tokens Adaptive decoupling into common sparse components and common residual components .

[0090] Step S2 includes the following steps: Step S2-1: Calculate the amplitude threshold based on the shared sparse budget. Calculate the input word embedding amplitude threshold This ensures that the proportion of elements in the input lexical embedding with an absolute value less than the threshold conforms to the sparse budget.

[0091] Step S2-2: Extract common sparse components and use amplitude thresholding. Embedding of input tokens Each element in After filtering, common sparse components are obtained. Its calculation formula is expressed as: ; The element-level sparsity operator function is specifically defined as follows: .

[0092] Steps 2-3: Extract common residual components Embed the input tokens Subtract common sparse components The common residual components are obtained. The calculation formula is as follows: .

[0093] Step S3, for common sparse components Perform two-layer bitmasking and quantization to obtain the common bitmask matrix. and common sparse quantization value Simultaneously, using the low-rank basis matrix of the bond... Sum-valued low-rank basis matrix For the residual components respectively Projection and quantization are performed to obtain the quantized values ​​of the bond residuals. Sum residual quantized value .

[0094] Step S3 includes the following steps: Step S3-1: Construct common sparse coding and apply it to the common sparse components. Perform a bitmasking operation to obtain the outer bitmask. The formula for recording the positions of non-zero elements in the sparse component is: ; in, This is an indicator function.

[0095] Step S3-2, for common sparse components Perform INT8 uniform quantization to obtain common sparse quantization values. Then, a bitmasking operation is performed on the quantized result to obtain the inner bitmask. Its formula is: ; ; in, This indicates that INT8 uniform quantization is being performed.

[0096] Step S3-3: Construct the key residual encoding using the low-rank key basis matrix. Common residual components Projecting to a lower-dimensional subspace yields bond intermediate states. Then, it is uniformly quantized using INT4 to obtain the bond residual quantization value. Its formula is: ; ; in, This indicates that INT4 uniform quantization is being performed. This is the left low-rank basis matrix obtained from offline decomposition.

[0097] Step S3-4: Construct value residual encoding using a value low-rank basis matrix. Common residual components Projecting onto a lower-dimensional subspace yields the value intermediate state. Then, perform INT4 uniform quantization on it to obtain the residual quantized value. Its formula is: ; .

[0098] Step S4: Construct a cache module and encode the common structured code. With independent structured coding It is stored in video memory to replace the traditional KV matrix storage.

[0099] Step S4 includes the following steps: Step S4-1: Store the common structured code. Allocate a common storage area in the video memory to store the common structured code. ; where the outer bit mask and inner bitmask It uses a bit-packing format for compressed storage.

[0100] Step S4-2: Store the independent structured code. Allocate an independent storage area in the video memory to store the independent structured code. .

[0101] Step S4-3: Construct a cache index, establish a hierarchical mapping relationship, and associate the common structured encoding with the independent structured encoding to the first level of the large language model. This layer serves as the sole persistent representation of the key-value data within that layer, replacing the original KV matrix storage.

[0102] Step S5: Construct a dynamic reconstruction module. In each decoding step of the inference phase, call the custom fusion operator component to encode the common structure. Perform decompression and dequantization operations to obtain the reconstructed common... And using the weight matrix and By projecting separately, the sparse contribution is obtained. and .

[0103] Step S5 includes the following steps: Step S5-1: Reconstruct the common sparse components, start the custom fusion operator component, and use the hardware built-in instructions to directly extract the common bitmask. The non-zero memory offset is quickly calculated, and the common sparse quantization value is read based on the offset. And perform the inverse quantization operation to obtain the reconstructed common... .

[0104] Step S5-2: In the custom fusion operator component, utilize the static weight matrix. and For the reconstructed public , obtain sparse contribution and Its formula is: ; .

[0105] Step S6: In the custom fusion operator component, perform independent structured encoding. Perform dequantization and utilize the low-rank basis matrix of the bond. Sum-valued low-rank basis matrix The residual contributions were obtained by projecting the data separately. and .

[0106] Step S6 includes the following steps: Step S6-1: In the custom fusion operator component, read the quantized value of the key residual. Sum residual quantized value And perform dequantization to obtain the bond intermediate states respectively. Sum of intermediate states .

[0107] Step S6-2, using the low-rank basis matrix of the bond Sum-valued low-rank basis matrix The contribution of the bond residuals was obtained. Sum residual contribution Its formula is: ; ; in, and This is the right low-rank basis matrix obtained from offline decomposition.

[0108] Step S7: After calculating the sparse contribution and residual contribution, they are added in real time within the operator to reconstruct the transient key-value tensor. and And release it immediately after the attention calculation is complete.

[0109] Step S7 includes the following steps: Step S7-1: In the custom fusion operator component, the bond sparsity contribution obtained in step S5 is processed. Compared with the bond residual contribution obtained in step S6 Simultaneously, the sparse contribution is made to the value obtained in step S5. Residual contribution compared to the value obtained in step S6 Its formula is: ; .

[0110] Step S7-2, convert the transient key tensor and transient value tensor The input attention calculation module performs the dot product operation and releases it immediately after the attention output calculation is completed at the current time step.

[0111] Please refer to the definition of each expression in the invention content and specific embodiments for the definitions of the above expressions.

[0112] Using publicly available large language models (including Llama-3-8B-1M, GLM-4-9B-1M, Llama-3-8B, etc.) as experimental subjects, the results were validated on multiple authoritative evaluation datasets. For example, in tests on LongBench (which includes tasks such as long text summarization and question answering), the present invention achieved a 3-fold reduction in KV cache memory usage while maintaining an average performance score less than 1% lower than the full-precision baseline model, essentially achieving lossless compression. In tests on the GSM8K mathematical inference dataset, the inference accuracy of the present invention was essentially on par with the baseline model. Furthermore, regarding inference latency, thanks to hardware optimization of the custom fusion operator component, the present invention also demonstrated a competitive advantage in end-to-end inference speed for long sequences (such as 64K words).

[0113] The aforementioned evolutionary search submodule, singular value decomposition submodule, pre-filling decoupling module, attention calculation module, adaptive decoupling unit, common sparse component masking and quantization processing unit, common residual component projection and quantization processing unit, common structured coding unit, compression unit and independent structured coding unit, custom fusion operator component, projection component, common data decompression operator, common data dequantization operator, independent data dequantization operator, transient key tensor operator, transient value tensor operator, common sparse component projection unit, key intermediate state matrix projection unit, and value intermediate state matrix projection unit can all adopt applicable functional modules and algorithms in the prior art, or adopt functional modules and algorithms in the prior art and construct and implement them using conventional technical means.

[0114] The above embodiments are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, all equivalent changes or modifications made in accordance with the spirit of the present invention still fall within the patent scope of the present invention.

Claims

1. A key-value caching optimization system for reducing the memory overhead of long text inference in large language models, characterized in that, The system includes an offline optimization module, a pre-filled decoupling module, a caching module, and a dynamic reconstruction module. The offline optimization module is used to determine the optimal compression parameters for each layer of the large language model and to perform offline decomposition of the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, and obtains a shared sparse budget, independent key rank values ​​and value rank values. The static key weight matrix and static value weight matrix of the attention mechanism of each layer of the large language model are respectively decomposed using the singular value decomposition submodule, and the corresponding key low-rank basis matrix and value low-rank basis matrix are obtained. The pre-filled decoupling module is used to adaptively decouple the input lexical embeddings into sparse components and residual components based on a shared sparse budget, and generate structured coded data from the sparse components and residual components. The cache module is used to store the structured coded data generated by the pre-filled decoupling module, which replaces the KV matrix. The dynamic reconstruction module is used to reconstruct structured encoded data into transient KV tensors in real time during the inference phase using custom operators, and releases them immediately after the attention calculation is completed.

2. The KV caching optimization system for reducing the memory overhead of long text inference in large language models according to claim 1, characterized in that, The pre-filled decoupling module includes an adaptive decoupling unit, a common sparse component masking and quantization processing unit, a common residual component projection and quantization processing unit, a common structured coding unit, a compression unit, and an independent structured coding unit. The adaptive decoupling unit is used to adaptively decouple the input lexical embeddings into common sparse components and common residual components based on a shared sparse budget. The common sparse component masking and quantization processing unit is used to perform double-layer bit masking and quantization processing on the common sparse components respectively, and obtain the common bit mask matrix and common sparse quantization value accordingly. The common residual component projection and quantization processing unit is used to project and quantize the common residual component sequentially using the bond low-rank basis matrix and the value low-rank basis matrix, respectively, to obtain the quantized value of the bond residual and the quantized value of the value residual. The common structured coding unit is used to perform common structured coding on the common bitmask matrix and common sparse quantization values ​​to generate common structured coded data. The compression unit is used to compress common structured coded data; Independent structured coding units are used to perform independent structured coding on the quantized values ​​of key residuals and value residuals to generate independent structured coded data.

3. The KV caching optimization system for reducing the memory overhead of long text inference in large language models according to claim 2, characterized in that, The dynamic reconstruction module includes a custom fusion operator component and a projection component; the custom fusion operator component includes a common data decompression operator, a common data inverse quantization operator, an independent data inverse quantization operator, a transient key tensor operator, and a transient value tensor operator; the projection component includes a common sparse component projection unit, a key intermediate state matrix projection unit, and a value intermediate state matrix projection unit; The public data decompression operator is used to perform decompression processing on compressed public structured coded data; The public data dequantization operator is used to perform dequantization operations on public structured coded data to obtain the reconstructed public sparse components; The independent data dequantization operator is used to perform dequantization operations on independent structured coded data to obtain the reconstructed key intermediate matrix and value intermediate matrix; The common sparse component projection unit is used to project the common sparse component using the static bond weight matrix and the static value weight matrix respectively, and obtain the bond sparse contribution and the value sparse contribution accordingly. The bond intermediate state matrix projection unit is used to project the bond intermediate state matrix using the low-rank basis matrix of the bond to obtain the bond residual contribution. The key intermediate state matrix projection unit is used to project the value intermediate state matrix using a value low-rank basis matrix to obtain the value residual contribution. The transient bond tensor operator is used to add the bond sparse contribution and the bond residual contribution in real time to obtain the reconstructed transient bond tensor; The transient value tensor operator is used to add the sparse contribution of the value to the residual contribution of the value in real time to obtain the reconstructed transient value tensor.

4. A key-value caching optimization method for reducing the GPU memory overhead of long text inference in large language models, characterized in that, The method includes the following steps: Step 1: Construct an offline optimization module to determine the optimal compression parameters for each layer of the large language model and perform offline decomposition on the static weights. The offline optimization module includes an evolutionary search submodule and a singular value decomposition submodule. The evolutionary search submodule is used to determine the compression configuration parameters for each layer of the large language model, obtaining a shared sparse budget, independent key-rank values, and value-rank values. The static key-weight matrix and static value-weight matrix of the attention mechanism of each layer of the large language model are decomposed using the singular value decomposition submodule, respectively, to obtain the corresponding low-rank key basis matrix and low-rank value basis matrix. Step 2: Construct a pre-filled decoupling module. Based on a shared sparse budget, the pre-filled decoupling module adaptively decouples the input lexical embeddings into common sparse components and common residual components. Step 3: Perform double-layer bitmasking and quantization on the common sparse components to obtain the common bitmask matrix and common sparse quantization value; use the bond low-rank basis matrix and the value low-rank basis matrix to project and quantize the common residual components in sequence to obtain the bond residual quantization value and the value residual quantization value. Step 4: Use common structured encoding for the common bitmask matrix and common sparse quantization values, and use independent structured encoding for the key residual quantization values ​​and value residual quantization values. Construct a video memory cache module so that the common structured encoded data and independent structured encoded data replace the KV matrix and are stored in the video memory cache module. Step 5: Construct a dynamic reconstruction module. In the dynamic reconstruction module, set up a custom fusion operator component. In each decoding step of the large language model inference stage, call the custom fusion operator component to perform decompression and dequantization operations on the common structured encoded data to obtain the reconstructed common sparse components. Then, the common sparse components are projected using the static key weight matrix and the static value weight matrix respectively, and the corresponding key sparse contribution and value sparse contribution are obtained. Step 6: Perform dequantization operation on the independent structured coded data using a custom fusion operator component to obtain the reconstructed key intermediate state matrix and value intermediate state matrix; and project the reconstructed key and value intermediate state matrices using the key and value low-rank basis matrices to obtain the corresponding key residual contribution and value residual contribution. Step 7: During the online long text inference process of the large language model, the key sparse contribution and key residual contribution are added in real time within the custom fusion operator component to obtain the reconstructed transient key tensor, and the value sparse contribution and value residual contribution are added to obtain the reconstructed transient value tensor. The transient key tensor and transient value tensor are released immediately after the attention calculation is completed.

5. The KV cache optimization method for reducing the memory overhead of long text inference in large language models according to claim 4, characterized in that, Corresponding to the large language model Layer, assuming: shared sparse budget is The key rank value is The rank value is The compression configuration parameter recipe is as follows , The static key weight matrix is: The static value weight matrix is Step 1 includes the following sub-steps: Step 1-1, let: In the evolutionary search submodule, The population size is The number of iterations is The rate of variation Crossover rate ; The The formula is ;Include The population set of the candidate formulations is , The current globally optimal formula is ; Assignment , , , ,initialization and ; Step 1-2, Perform differential mutation operation: In each generation iteration, traverse the current population. For the first One recipe From the population Three different recipes are randomly selected. The following differential mutation formula is used to generate candidate formulations: ； In the formula: for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for population The first in One recipe, =1,2,…,P; for Candidate formulations for mutations; Steps 1-3, perform crossover operation: for the first... One recipe With probability Select from variant candidate formulations The value in the middle, thus generating the offspring recipe, let For the corresponding number One recipe Offspring formula; Steps 1-4, apply projection constraints: adjust the offspring recipe The parameter values ​​are projected to a preset range of valid values ​​to ensure a shared sparse budget. And rank value If the integer is positive, a feasible offspring formula is obtained; Steps 1-5, Fitness Evaluation: Apply feasible progeny formulas to the large language model, perform inference tests on the validation set, calculate validation loss or accuracy, and use the validation loss or accuracy as the fitness score. Steps 1-6, Selection and Update: This involves generating a set of recipes for all offspring. with parental population Merge, sort by fitness score and retain the top score. The optimal recipe is used as the next generation population, and the globally optimal recipe is updated simultaneously. ; Steps 1-7, for the static key weight matrix Perform key singular value decomposition; the decomposition formula is as follows: ； In the formula: Static key weight matrix The left singular vector matrix; Static key weight matrix The right singular vector matrix; For containing static key weight matrix A diagonal matrix of singular values; Sort the key singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The _n_ key singular values ​​and their corresponding key singular vectors are used to construct the low-rank basis matrix of the keys according to the following formula: ； ； In the formula: To sort by first The left key singular vector matrix corresponding to each key singular value; To sort by first The right-key singular vector matrix corresponding to each key singular value; For including the previous A diagonal matrix of singular values ​​of each key; The left-hand low-rank basis matrix; ; The right-click low-rank basis matrix; ; For the hidden layer dimension; For projection dimensions; Steps 1-8: Adjust the static value weight matrix. The singular value decomposition of the execution value is performed, and the decomposition formula is as follows: ； In the formula: Static value weight matrix The left singular vector matrix; Static value weight matrix The right singular vector matrix; Static value weight matrix A diagonal matrix of singular values; Sort the singular values ​​from largest to smallest, and extract the first few elements in the sorted order. The singular values ​​and their corresponding singular vectors are used to construct a low-rank basis matrix of values ​​according to the following formula: ； ； In the formula: To sort by first The left-value singular vector matrix corresponding to the singular values ​​of each singular value; To sort by first The rvalue singular vector matrix corresponding to the singular values ​​of each singular value; For including the previous A diagonal matrix with singular values; It is an lvalued low-rank basis matrix; ; It is an rvalued low-rank basis matrix; .

6. The KV cache optimization method for reducing the GPU memory overhead of long text inference in large language models according to claim 4, characterized in that, Step 2 includes the following sub-steps: Step 2-1, corresponding to the first step of the large language model. Layer, let: input lexical embedding be , The amplitude threshold is Shared sparse budget is ; Based on shared sparse budget Calculate the input word embedding amplitude threshold This ensures that the proportion of elements in the input lexical embedding whose absolute value is less than the threshold conforms to the sparse budget. Step 2-2: Extract common sparse components and use amplitude thresholding. Embedding of input tokens Each element in the dataset is filtered to obtain the common sparse component. The formula for calculating the common sparse component is as follows: ； In the formula: For common sparse components; ; Represents element-level sparsification operator functions; Batch size; For context length; For the hidden layer dimension; The function expression is as follows: ； In the formula: Embedding of input tokens Elements in; Steps 2-3: Extract the common residual component. Subtract the common sparse component from the input token embedding to obtain the common residual component. The formula for calculating the common residual component is: ； In the formula: For common residual components; .

7. The KV cache optimization method for reducing the memory overhead of long text inference in large language models according to claim 4, characterized in that, Step 3 includes the following sub-steps: Step 3-1, corresponding to the first step of the large language model. Layer, let: input lexical embedding be , The amplitude threshold is Shared sparse budget is The common sparse component is The key rank value is The rank value is ; For common sparse components Perform bitmasking operations to obtain the outer bitmask, which records the positions of non-zero elements of the common sparse components. The formula for calculating the outer bitmask is as follows: ； In the formula: for The outer bitmask; ; Batch size; For context length; For the hidden layer dimension; For indicator functions; Step 3-2, for common sparse components Perform INT8 uniform quantization to obtain a common sparse quantized value, and then perform a bitmasking operation on the common sparse quantized value to obtain the inner bitmask. The formula for calculating the inner bitmask is as follows: ； ； In the formula: For common sparse quantization values; This indicates that INT8 uniform quantization is being performed; for Inner bitmask; ; Step 3-3: Construct the bond residual code. Project the common residual components onto the low-dimensional subspace using the low-rank basis matrix of the bonds to obtain the bond intermediate state matrix. Then, perform INT4 uniform quantization on the bond intermediate state matrix to obtain the quantized value of the bond residual. The formula for calculating the quantized value of the bond residual is as follows: ； ； In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​the bond matrix; ; For common residual components; The left-hand low-rank basis matrix; This represents the INT4 uniform quantization function; Steps 3-4 involve constructing value residual codes. Common residual components are projected onto a low-dimensional subspace using a low-rank value basis matrix to obtain a value intermediate state matrix. This matrix is ​​then subjected to INT4 uniform quantization to obtain the value residual quantized values. The formula for calculating the value residual quantized values ​​is as follows: ； ； In the formula: Quantized values ​​of bond residuals; The intermediate state matrix is ​​a value. ; It is an left-valued low-rank basis matrix.

8. The KV cache optimization method for reducing the memory overhead of long text inference in large language models according to claim 7, characterized in that, Step 4 includes the following sub-steps: Step 4-1, corresponding to the first step of the large language model. The layer allocates a common storage area in the video memory to store the following common structured encoded data: ; where the outer bit mask and inner bitmask Compressed storage is performed using a bit-packed format; Step 4-2: Allocate a separate storage area in the video memory to store the following independent structured coded data: ; Step 4-3: Construct a cache index, establish a hierarchical mapping relationship, and associate the common structured encoded data with the independent structured encoded data to the first level of the large language model. This layer serves as the sole persistent representation of the key-value data within that layer, replacing the original KV matrix storage.

9. The KV cache optimization method for reducing the memory overhead of long text inference in large language models according to claim 4, characterized in that, Step 5 includes the following sub-steps: Step 5-1, corresponding to the first step of the large language model. In the first layer, a custom fusion operator component is started. The non-zero memory offset is calculated directly from the common bit mask using the hardware built-in instructions. The common sparse quantization value is read based on the non-zero memory offset and the inverse quantization operation is performed to obtain the reconstructed common sparse components. Step 5-2: In the custom fusion operator component, perform sparse matrix multiplication operations on the static key weight matrix and the static value weight matrix with the reconstructed common sparse components to obtain the key sparse contribution and value sparse contribution, respectively. The calculation formulas are as follows: ； ； In the formula: Static key weight matrix; Static value weight matrix; For the reconstructed common sparse components; Contribute to bond sparsity; This contributes to the sparse value.

10. The KV cache optimization method for reducing the memory overhead of long text inference in large language models according to claim 4, characterized in that, Step 6 includes the following sub-steps: Step 6-1, corresponding to the first step of the large language model. In the custom fusion operator component, the key residual quantized value and the value residual quantized value are read and dequantized to obtain the reconstructed key intermediate state matrix and value intermediate state matrix, respectively. Step 6-2: Perform dense matrix multiplication on the low-rank bond basis matrix and the low-rank value basis matrix, corresponding to the reconstructed intermediate bond matrix and the intermediate value matrix, to obtain the bond residual contribution and the value residual contribution, calculated as follows: ； ； In the formula: Contribution to bond residuals; ; Contribution to residual values; ; Batch size; For context length; For projection dimensions; This is the reconstructed bond intermediate state matrix; This is the reconstructed intermediate state matrix; The right-click low-rank basis matrix; ; It is an rvalued low-rank basis matrix; ; The key rank value; The value is the rank value.

Images