Large language model KV cache compression method based on interlayer fusion

Abstract

The invention belongs to the technical field of large language models, and particularly relates to a large language model KV cache compression method based on interlayer fusion. The method comprises the following steps: inputting a to-be-processed text into a large language model to obtain a KV cache; the compression ratio of each layer of the large model is calculated, and preliminary SVD compression is carried out on KV cache of different layers of the large model; the KV cache subjected to preliminary compression is subjected to block division, and a plurality of blocks are obtained; calculating the attention score of each block; s blocks with the highest attention score are selected, and SVD reconstruction is carried out on the selected blocks; constructing a cache pool to store the reconstructed S blocks; dynamically updating the cache pool; splicing the S blocks in the cache pool to obtain a KV cache, and completing the compression of the KV cache; according to the method, the model is ensured to maintain stable prediction accuracy under diversified task scenes such as text generation and question and answer systems, and balance between storage efficiency and performance is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of large language models, and in particular relates to a large language model KVcache compression method based on inter-layer fusion. Background Art

[0002] During the reasoning process of large language models (LLMs), the storage and transmission overhead of KV Cache (key-value cache) increases exponentially with the increase of model scale and sequence length, which makes efficient cache compression technology a key link in optimizing reasoning efficiency. Although the current mainstream unified compression strategy compresses the KV Cache of all layers through a fixed compression rate or a single algorithm, which can reduce video memory usage and bandwidth pressure to a certain extent, this "one-size-fits-all" compression paradigm ignores the significant differences in features at different levels in the Transformer architecture. Specifically, the attention layer at the low level of the model mainly processes fine-grained local semantics and grammatical information (such as word order and phrase structure). Its feature vector has high information density and low redundancy. If too high a compression rate is used, it is easy to cause the loss of key details, thereby affecting the feature extraction accuracy of subsequent layers; while the attention mechanism at the high level of the model (close to the output layer) focuses more on abstract semantics and global associations. Its feature vector has high information redundancy and compressible space. Using a conservative low compression rate will result in a waste of storage resources. Figure 1 As shown, higher layers focus on fewer tokens, allowing for higher compression rates without sacrificing quality. However, lower layers focus on more tokens and are highly sensitive to compression rates. This gradient distribution of compression sensitivity across layers creates a dilemma for unified compression strategies in practical applications: over-compression of lower layers destroys fundamental semantic units, leading to cumulative errors; while under-compression of higher layers hinders the full potential of video memory optimization. Therefore, developing a differentiated compression framework based on hierarchical dynamic awareness will become an important research direction to overcome existing technical bottlenecks.

[0003] To reduce the memory and computational burden of KV caching, many studies have proposed methods for compressing KV caches. These methods typically select a subset of important tokens and retain only their keys and values ​​for approximate attention calculations. There are two main approaches to selecting important tokens: one is to select according to a fixed rule or pattern (for example, a local attention mechanism only retains the KV cache of the initial and most recent tokens), and the other is to select based on attention weights—attention weights reflect the importance of each token in calculating attention.

[0004] However, most compression methods permanently delete (i.e., purge from the cache) tokens that are considered "unimportant." Once deleted, these tokens will not be reused in subsequent decoding steps. While this approach can reduce storage requirements and computational overhead, it ignores an important fact: the importance of tokens changes dynamically during the decoding process. For example, a token that is unimportant in the current decoding step may become very important in the subsequent generation step, and vice versa.

[0005] These traditional compression methods are often "greedy," removing seemingly unimportant tokens based solely on their position or attention weight at the current decoding step. This approach fails to capture the dynamic nature of token importance. This flaw can lead to reduced accuracy in model generation and even compromise overall output quality.

[0006] SVD-based compression is an emerging research direction for KV cache. SVD decomposes the matrix into the product of three sub-matrices M = UΣV T SVD compression achieves data compression by extracting the eigenvectors corresponding to the primary singular values ​​and ignoring the secondary components. Compared to previous methods, SVD compression can preserve global semantic features through low-rank approximation, achieving higher compression rates. However, SVD-based compression requires rebuilding the KV cache during inference, which still limits storage during inference.

[0007] In summary, a new KV cache compression method for large language models is urgently needed to address the problems of different compression sensitivities at different layers and memory limitations during the inference phase, thereby reducing the memory overhead of the KV cache for large language models. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the present invention proposes a large language model KVcache compression method based on inter-layer fusion, which includes:

[0009] S1: Input the text to be processed into the large language model to obtain the KV cache;

[0010] S2: Calculate the compression ratio of each layer of the large model;

[0011] S3: Perform SVD compression on the KV caches at different layers of the large model based on the compression ratio of each layer of the large model to obtain a preliminarily compressed KV cache.

[0012] S4: Divide the KV cache after preliminary compression into multiple blocks and calculate the attention score of each block;

[0013] S5: Select the S blocks with the highest attention scores and perform SVD reconstruction on the selected blocks;

[0014] S6: Build a cache pool to store the reconstructed S blocks; dynamically update the cache pool;

[0015] S7: Concatenate the S blocks in the cache pool to obtain the KV cache and complete KV cache compression.

[0016] Preferably, the process of calculating the compression ratio of each layer of the large model includes:

[0017] Calculate the cosine similarity between the input and output of each layer of the model;

[0018] The cosine similarity clusters are divided into three groups, namely G1, G2 and G3; the absolute value of cosine similarity in group G3 is the largest, and the absolute value of cosine similarity in group G1 is the smallest;

[0019] Set the compression ratio of the G3 group and the overall compression ratio, and calculate the compression ratio of each layer based on the compression ratio of the G3 group and the overall compression ratio.

[0020] Furthermore, the formula for calculating cosine similarity is:

[0021]

[0022] Among them, cos_sim (i) Represents the cosine similarity of the input and output of the model layer i as a whole, prompt represents the input sentence sequence, represents the cosine similarity between the input and output of the jth token in the i-th layer, and len(·) represents the calculation length.

[0023] Furthermore, the formula for calculating the compression rate of each layer based on the compression rate of the G3 group and the overall compression rate is:

[0024]

[0025] Among them, b i represents the compression rate of the i-th layer of the model, b init represents the overall compression ratio, p represents the compression ratio of the G3 group, len(·) represents the calculation length, n layer Indicates the number of layers in the model.

[0026] Preferably, the process of partitioning the KV cache includes:

[0027] Perform block operations on the key state tensor of the KV cache to obtain multiple blocks;

[0028] The key state tensors in each block are averaged to obtain the representative state tensor of each block.

[0029] Furthermore, the attention score of each block is calculated as:

[0030]

[0031] Among them, score represents the attention score of the block, query represents the query vector, avg_key represents the representative state tensor, D represents the dimension of the attention head, and softamx(·) represents the softmax function.

[0032] Preferably, the cache pool is represented as:

[0033] C t ={(c i ,t i )|t i ∈[tT,t)}

[0034] Among them, C t represents the buffer pool of time step t, c i represents the i-th reconstructed block, t i represents the time step after the i-th reconstructed block is visited, and T represents the length of the time window.

[0035] Preferably, the process of dynamically updating the cache pool includes:

[0036] Calculate the attention score of the current query vector and the key state tensor of the block, and select the block indexes corresponding to the S blocks with the highest attention scores to form a block index set;

[0037] Check whether the block index set of the current query vector hits the cache pool; if so, directly reuse the cache pool data and update the access time; otherwise, perform SVD reconstruction on the block to obtain a new block;

[0038] The cache pool removes expired blocks and adds new blocks after reconstruction.

[0039] The beneficial effects of the present invention are as follows: in order to achieve efficient and accurate KV cache compression, the present invention adopts an innovative method that combines cross-layer singular value decomposition (SVD) with an intelligent blocking strategy. First, based on the differences in the sensitivity of each layer to compression processing, a customized compression threshold is assigned to it, and the traditional SVD technology is extended to cross-layer applications. While achieving substantial data compression, the global features between layers are retained to the greatest extent, avoiding the loss of key information. Secondly, by dividing the KV cache into multiple data blocks, the representative data building blocks of each sub-block are extracted to represent the KV cache. In the inference decoding stage, only the key block representative KV screened out by the attention mechanism is subjected to SVD reconstruction calculation, which significantly reduces the computational overhead of the reconstruction process. Through the above-mentioned dual optimization mechanism, the present invention effectively compresses the KV cache storage occupancy while ensuring that the model maintains a stable prediction accuracy in diverse task scenarios such as text generation and question-answering systems, thereby achieving a balance between storage efficiency and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a visualization diagram of the attention tokens at different layers of the model in this invention;

[0041] Figure 2 This is a flow chart of the large language model KV cache compression method based on inter-layer fusion in the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The present invention proposes a large language model KV cache compression method based on inter-layer fusion, such as Figure 2 As shown, the method includes the following contents:

[0044] S1: Input the text to be processed into the large language model to obtain the KV cache.

[0045] The text to be processed is input into the Large Language Model (LLM). During the generative LLM reasoning process, to avoid repeated calculations, the model needs to store the key (Key) and value (Value) vectors of all previous tokens in the KV cache (key-value cache) to facilitate fast self-attention calculations in the decoding phase.

[0046] S2: Calculate the compression ratio of each layer of the large model.

[0047] For each attention layer of the model, the cosine similarity between the input and output is calculated. Different compression rates are assigned to different layers based on the cosine similarity results. The compression rate is reflected in the selection of the singular values ​​mentioned above. The greater the compression rate, the fewer singular values ​​are retained, and the smaller the compression rate, the greater the singular values ​​retained. The formula for calculating the cosine similarity between the input and output of each layer of the model is:

[0048]

[0049] Among them, cos_sim (i) Represents the cosine similarity of the input and output of the model layer i as a whole, prompt represents the input sentence sequence, represents the cosine similarity between the input and output of the jth token in the i-th layer, and len(·) represents the calculation length.

[0050] The cosine similarity clustering is divided into three groups, namely G1, G2 and G3; among them, the absolute value of cosine similarity in group G3 is the largest, and the absolute value of cosine similarity in group G1 is the smallest.

[0051] G1,G2,G3←KMeans(cos_sim (i) )

[0052] Set the compression rate of the G3 group and the overall compression rate, and calculate the compression rate of each layer based on the compression rate of the G3 group and the overall compression rate, expressed as:

[0053]

[0054] Among them, b i represents the compression rate of the i-th layer of the model, b init represents the overall compression rate, p represents the compression rate of the G3 group, and n layer Indicates the number of layers in the model.

[0055] The present invention quantifies the sensitivity of each layer to compression processing by accurately calculating the cosine similarity between the input and output of the attention layer: when the cosine similarity between the input and output is high, it means that the data of this layer has not changed significantly during the calculation process, and the effective information it carries is relatively small, thus having a high compression sensitivity and can be processed with a higher compression rate; conversely, if the cosine similarity is low, it indicates that the data of this layer has undergone rich semantic changes during the calculation process, carrying a large amount of key information, and a smaller compression threshold is required to prevent information loss. The model is divided into three categories according to the above cosine similarity, and differentiated compression rates are tailored for the characteristics of each layer, ensuring that under the premise of a fixed overall compression rate, the compression threshold can be intelligently and efficiently allocated based on the unique attributes of the information carried by each layer.

[0056] S3: Perform SVD compression on the KV caches of different layers of the large model according to the compression ratio of each layer of the large model to obtain a preliminarily compressed KV cache.

[0057] By assigning different compression rates to different layers of the model and mapping them to the number of singular values ​​retained in the SVD, dynamic compression between different layers of the SVD can be achieved. Specifically:

[0058] Assume that the KV cache of a set of layers of the selected model is Then perform SVD decomposition to obtain:

[0059]

[0060] Where r is determined by the compression ratio, represents the number of singular values ​​that need to be retained, G is the number of layers in each group, and d represents the dimension of KVcache.

[0061] Further analysis:

[0062]

[0063] Where A is the shared SVD recovery matrix for each group, It is the KV cache after SVD compression at each layer.

[0064] The present invention uses an idea based on SVD matrix decomposition to achieve efficient compression of KV cache, achieving efficient and nearly lossless compression effects. Compared with traditional quantization methods, which often require a large amount of computing resources to be retrained to restore model performance during the compression process, and rely on attention scores to expel tokens to achieve compression, but inevitably cause problems such as global information loss, the present invention has significant advantages. This solution does not require any retraining steps and directly processes the KV cache, fundamentally saving training costs and time. At the same time, it abandons the traditional way of compressing by token dimensions and instead uses SVD matrix decomposition technology to perform overall compression processing on the KV cache, which can not only achieve a high compression rate, but also accurately capture the global semantic information of the model, ensuring that key information is not missed during the model reasoning process, and achieving a perfect balance between storage efficiency and semantic expression capabilities.

[0065] The present invention targets KV cache compression scenarios and introduces a multi-layer SVD method based on traditional SVD compression technology, significantly improving compression efficiency by constructing an inter-layer semantic information sharing mechanism. Specifically, cross-layer SVD technology can project KV cache data of different levels into an inter-layer shared subspace, which can capture cross-layer semantic correlation features, so that the model can still retain complete global semantic information at low compression rates. Compared with the traditional "separate SVD compression per layer" mode, the latter lacks inter-layer semantic collaboration capabilities and can only rely on independent compression of each layer, resulting in the problem of inter-layer semantic fracture when the compression rate is improved (such as above a certain threshold), which ultimately causes a significant decrease in model accuracy. The present invention adopts the idea of ​​multi-layer SVD fusion and multi-layer shared SVD recovery matrix to achieve the construction of a cross-layer shared subspace, realizing efficient transmission and fusion of inter-layer semantic information during the compression process. Under the same compression rate conditions, the model accuracy can be kept stable, breaking through the performance bottleneck of traditional single-layer compression, and providing a new technical path with both efficiency and accuracy for the lightweight deployment of large-scale language models.

[0066] S4: Divide the KV cache after preliminary compression into multiple blocks; calculate the attention score of each block.

[0067] Although the current KV cache compression method based on singular value decomposition (SVD) performs well in static compression efficiency, its core flaw is that it does not fully consider the computational cost in dynamic reasoning scenarios. The traditional solution reduces the high-dimensional KV cache (dimension is d×n) to a low-dimensional space (d×k, k<<n) through a projection matrix. Although it significantly reduces the storage space occupancy, it causes a secondary problem of reverse dimensional expansion in the autoregressive decoding stage - each time a new token is generated, the compressed matrix needs to be fully rebuilt and restored, resulting in a surge in video memory bandwidth consumption (up to O(dn) level) and the introduction of additional computational delays. To address this systemic bottleneck, the present invention proposes a block dynamic reconstruction mechanism. First, a block partitioning strategy is adopted to decompose the KV cache into m continuous sub-blocks. A learnable block marker generator is used to extract the block representative KV for each sub-block. Specifically, the key state tensor of the KVcache is partitioned to obtain multiple blocks. Specifically:

[0068] Assume the original Key state tensor is:

[0069]

[0070] Where B is the batch size, H is the number of attention heads, L is the total sequence length, and D is the dimension of each head.

[0071] The block operation is as follows:

[0072] Key_states_block=Reshape(Key_states[:,:,:C*K],(B,H,C,K,D))

[0073] Among them, C is the number of blocks, K is the block size, and the size after blocking becomes B*H*C*K*D.

[0074] The key state tensors in each block are averaged to obtain the representative state tensor of each block, which is expressed as:

[0075]

[0076] Finally got

[0077] The attention score of each block is calculated based on its representative state tensor. The process is as follows:

[0078] Let the query vector be:

[0079]

[0080] Where B is the batch size, H is the number of attention heads, and D is the head dimension.

[0081] The attention score of each block is calculated as:

[0082]

[0083] Among them, score represents the attention score of the block, query represents the query vector, avg_key represents the representative state tensor, D represents the dimension of the attention head, and softmax(·) represents the softmax function.

[0084] S5: Select the S blocks with the highest attention scores and perform SVD reconstruction on the selected blocks.

[0085] Based on the Top-k filtering mechanism, the most relevant sub-blocks of the current reasoning context are dynamically identified, that is, the scores of each head are sorted and the indexes of the top S blocks are selected:

[0086] top_indices[b,h]=argtopk c∈{1,…,C} (score[b,h,c]) S

[0087] Among them, argtopk returns the index of the S blocks with the highest scores, and the result shape is B*H*S; extract the selected block and extract the corresponding block from the block data according to the index:

[0088] selected_chunks=key_states_block[top_indices]

[0089] in,

[0090] Perform SVD reconstruction on the selected block:

[0091] reconstructed_chunk[b,h,c]=U (b,h,c) [:,:,r]*diag(S (b,h,c) [:r])*V (b,h,c) [:,:,r] T

[0092] Among them, reconstructed_chunk[b,h,c] represents the result after the selected block is restored, U (b,h,c) [:,:,r] represents the top-r left singular vectors of SVD, S (b,h,c) [:r] represents the top-r singular values ​​of SVD, V (b,h,c) [:,:,r] T Represents the top-r right singular vectors of SVD.

[0093] In the field of KV cache compression optimization, the present invention proposes an innovative hybrid compression strategy to address the information loss and performance degradation problems of traditional compression technologies. The traditional permanent eviction method based on attention score achieves compression by directly removing low-scoring tokens. Although it can reduce the amount of data, it will cause irreversible loss of key semantic information, resulting in a significant decline in the performance of the model during the inference phase. To solve this problem, the present invention organically combines the block-level TopK sparsification method with the SVD compression technology to construct a dynamic information recovery mechanism. During the compression process, the SVD compression technology performs dimensionality reduction on the overall data to achieve efficient compression. The block-level TopK sparsification method dynamically restores data by block during the decoding phase. The two work together to ensure compression efficiency and reserve the necessary foundation for subsequent information recovery. In particular, the solution has dynamic adaptability and can flexibly adjust the information recovery strategy based on real-time changes in attention scores. During the dynamic generation process of model inference, when the attention score changes, the system can accurately identify the information fragments that need to be restored and reconstruct the compressed data in a timely manner, effectively avoiding the problem of accuracy decline caused by permanent loss of information, achieving dual guarantees of compression efficiency and model performance, and significantly improving the stability and reliability of the model in various application scenarios.

[0094] The present invention divides the KV cache data into multiple sub-blocks in the pre-filling stage, extracts the core features of each block through mean aggregation operations, and generates streamlined block representative KVs. These block representative KVs significantly reduce the data volume while maintaining the key information of the data, laying the foundation for subsequent efficient processing. Subsequently, the decoding stage performs TopK operations based on the block representative KV, accurately screening out the most valuable data blocks, marking them as requiring SVD reconstruction of key information, and dynamically restoring only the necessary data blocks, without the need to fully rebuild all KV cache data. This "block-based, on-demand recovery" strategy completely breaks the memory limitation shackles caused by the need for full recovery of traditional SVD compression, significantly reduces storage usage, and greatly improves the inference speed, providing solid technical support for the efficient deployment and smooth operation of large language models.

[0095] S6: Build a cache pool to store the reconstructed S blocks; dynamically update the cache pool.

[0096] The key values ​​of adjacent query vectors are highly similar, which is a characteristic of temporal locality. This paper proposes a time-window-based KV Cache update strategy. By retaining the reconstructed KV cache for a period of time and only updating the KV cache that the reconstruction missed, the number of reconstructions and the overhead can be partially reduced. Specifically:

[0097] Build a cache pool, that is, define a time window T to represent the step size of KV cache retention. For the current time step t, the cache pool retains all KV blocks reconstructed within the time window [tT, t). The cache pool is expressed as:

[0098] C t ={(c i ,t i )|t i ∈[tT,t)}

[0099] Among them, C t represents the buffer pool of time step t, c i represents the i-th reconstructed block, t i represents the time step at which the i-th reconstructed block is accessed, and T represents the length of the time window.

[0100] The process of dynamically updating the cache pool is as follows:

[0101] Set the current t i The query vector and the key vector corresponding to the block representative are used to calculate the attention score, and the S blocks with the highest attention scores are selected to represent the block set corresponding to the state tensor. The block indexes corresponding to the block set form a block index set, and the intersection operation is performed with the blocks in the cache pool to determine whether it is hit in the cache pool (the cache pool stores ti The specific process of high-frequency hit blocks before the moment is described in detail as follows.

[0102] Check whether the block index set of the current query vector hits the cache pool:

[0103]

[0104] If so, directly reuse the cache pool data and update the access time:

[0105]

[0106] Otherwise, perform SVD reconstruction on the block to obtain a new block New t .

[0107] The cache pool removes expired blocks (since the size of the cache pool is fixed, an LRU strategy can be used to delete the longest-missed blocks to make room for new hit blocks) and adds the newly reconstructed blocks:

[0108] C t+1 ={(c i ,t i )∈C t |t i ≥tT}∪{New t ,t)}

[0109] Aiming at the temporal locality feature of the query vector (i.e., the Key-Values ​​of adjacent queries are highly similar), the present invention proposes a time window-based KV Cache update strategy that effectively reduces the SVD reconstruction overhead through cache reuse and dynamic update mechanisms. The system records the last access time of the reconstructed KV Cache sub-block to form a cache pool. The cache pool only retains the KV blocks accessed within the most recent time window, thereby controlling the storage scale and retaining high-frequency usage data. This cache reuse strategy significantly reduces computing overhead and memory usage. By dynamically retaining high-value data and eliminating low-frequency data, the strategy can not only fully utilize temporal locality to improve efficiency, but also ensure the real-time and accuracy of model reasoning.

[0110] S7: Concatenate the S blocks in the cache pool to obtain the KV cache and complete KV cache compression.

[0111] Concatenate the S blocks in the buffer pool and reassemble the reconstructed blocks in the buffer pool into a key-state tensor:

[0112] reconstructed_key

[0113] =concat(reconstructed_chunk[b,h,1] key,...,reconstructed_chunk[b,h,S] key )

[0114] Similarly, the value state tensor is also reconstructed and combined

[0115] reconstructed_value

[0116] =concat(reconstructed_chunk[b,h,1] value ,...,reconstructed_chunk[b,h,S] value )

[0117] After KV cache compression is completed, the final text output can be obtained as follows.

[0118] (Remember reconstructed_key is K partial , let reconstructed_value be V partial )

[0119]

[0120] Output i Represents the final output of the i-th layer after compression, Attention_out i represents the output of the attention layer of the i-th layer after compression, d represents the dimension size of the KV cache itself, LN represents the layer normalization operation, and FFN represents the feedforward neural network.

[0121] In summary, the present invention introduces cosine similarity as a measurement standard to accurately quantify the sensitivity of each layer of the large model to compression processing. By calculating the cosine similarity of the input and output data of the attention layer, it is possible to effectively judge the amount of information change in the data of this layer during the calculation process, and then obtain different compression rates, further realize dynamic stratification, and customize differentiated compression strategies for different sensitivity levels, which significantly improves the accuracy and effectiveness of compression. The present invention performs overall dimensionality reduction processing on the KV cache through SVD matrix decomposition, effectively compresses the data scale without relying on retraining, and accurately captures and retains the global semantic information of the model, integrating the global information between layers. This technical solution not only avoids the problem of information loss caused by compression by token dimension in traditional compression methods, but also greatly reduces training costs and time. The present invention deeply integrates block-level TopK sparsification and SVD compression technology to construct a dynamic information recovery mechanism. This collaborative operation mode completely changes the problem of irreversible information loss caused by permanent expulsion of low-scoring tokens in traditional methods to achieve compression. The system can flexibly adjust information recovery strategies based on real-time changes in attention scores during inference, ensuring that even previously compressed data can be promptly reconstructed when it becomes critical information, achieving both improved compression efficiency and improved information integrity. This paper proposes a dynamic KV cache update strategy based on temporal locality. This "cache reuse + on-demand reconstruction" mechanism significantly reduces the computational overhead of repeated reconstructions. It also prevents unlimited cache expansion by automatically clearing expired data blocks, significantly improving the real-time performance and efficiency of model inference while reducing memory usage.

[0122] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large language model KV cache compression method based on inter-layer fusion, characterized in that: include: S1: Input the text to be processed into the large language model to obtain the KV cache; S2: Calculate the compression ratio of each layer of the large model; S3: Perform SVD compression on the KV caches at different layers of the large model based on the compression ratio of each layer of the large model to obtain a preliminarily compressed KV cache. S4: Divide the KV cache after preliminary compression into multiple blocks and calculate the attention score of each block; S5: Select the S blocks with the highest attention scores and perform SVD reconstruction on the selected blocks; S6: Build a cache pool to store the reconstructed S blocks; dynamically update the cache pool; S7: Concatenate the S blocks in the cache pool to obtain the KV cache and complete KV cache compression.

2. A large language model KV cache compression method based on inter-layer fusion according to claim 1, characterized in that: The process of calculating the compression ratio of each layer of a large model includes: Calculate the cosine similarity between the input and output of each layer of the model; The cosine similarity clusters are divided into three groups, namely G1, G2 and G3; the absolute value of cosine similarity in group G3 is the largest, and the absolute value of cosine similarity in group G1 is the smallest; Set the compression ratio of the G3 group and the overall compression ratio, and calculate the compression ratio of each layer based on the compression ratio of the G3 group and the overall compression ratio.

3. The large language model KV cache compression method based on inter-layer fusion according to claim 2 is characterized in that: The formula for calculating cosine similarity is: Among them, cos_sim (i) Represents the cosine similarity of the input and output of the model layer i, prompt represents the input sentence sequence, represents the cosine similarity between the input and output of the jth token in the i-th layer, and len(·) represents the calculation length.

4. The large language model KV cache compression method based on inter-layer fusion according to claim 2 is characterized in that: The formula for calculating the compression rate of each layer based on the compression rate of the G3 group and the overall compression rate is: Among them, b i represents the compression rate of the i-th layer of the model, b init represents the overall compression ratio, p represents the compression ratio of the G3 group, len(·) represents the calculation length, n layer Indicates the number of layers in the model.

5. The large language model KV cache compression method based on inter-layer fusion according to claim 1 is characterized in that: The process of partitioning the KV cache includes: Perform block operations on the key state tensor of the KV cache to obtain multiple blocks; The key state tensors in each block are averaged to obtain the representative state tensor of each block.

6. A large language model KV cache compression method based on inter-layer fusion according to claim 5, characterized in that: The attention score of each block is calculated as: Among them, score represents the attention score of the block, query represents the query vector, avg_key represents the representative state tensor, D represents the dimension of the attention head, and softmax(·) represents the softmax function.

7. The large language model KV cache compression method based on inter-layer fusion according to claim 1 is characterized in that: The cache pool is represented as: C t ={(c i ,t i )|t i ∈[t-T,t)} Among them, C t represents the buffer pool of time step t, c i represents the i-th reconstructed block, t i represents the time step after the i-th reconstructed block is visited, and T represents the length of the time window.

8. The large language model KV cache compression method based on inter-layer fusion according to claim 1 is characterized in that: The process of dynamically updating the cache pool includes: Calculate the attention score of the current query vector and the key state tensor of the block, and select the block indexes corresponding to the S blocks with the highest attention scores to form a block index set; Check whether the block index set of the current query vector hits the cache pool; if so, directly reuse the cache pool data and update the access time; otherwise, perform SVD reconstruction on the block to obtain a new block; The cache pool removes expired blocks and adds new blocks after reconstruction.

Images