A large language model key-value cache compression method and system based on semantic cluster center offset

Abstract

This invention relates to a key-value caching compression method and system for large language models based on semantic cluster center offset, belonging to the field of large language model inference acceleration technology. The method includes: a pre-filling stage, where semantic clustering is performed on the key matrix and cluster center vectors are extracted; the offset between the key vectors and the cluster centers is calculated; unstructured channel pruning and low-bit quantization are performed on the offsets; compressed data is stored in off-chip video memory; and cluster centers are stored in on-chip shared memory. In the decoding stage, compressed data is read from off-chip video memory; the offsets are dequantized to recover the data; approximate key vectors are reconstructed using the on-chip cluster centers; and attention calculation is performed. This invention maintains semantic integrity under high sparsity by transforming the compressed object from the original vector into an offset from the semantic cluster centers, eliminating the truncation and destruction of low-bit quantization by large numerical outliers, and achieving lossless superposition of pruning and quantization.

Description

Technical Field

[0001] This invention belongs to the field of Large Language Model (LLM) inference acceleration, and relates to a method and system for compressing key-value caches of large language models based on semantic cluster center offset. Background Technology

[0002] With the widespread application of LLM in scenarios such as long text analysis and multi-turn dialogues, the memory bottleneck in its inference process has become increasingly prominent. In the autoregressive decoding stage, to avoid redundant calculations of historical features, the inference engine needs to maintain a key-value (KV) cache. However, the storage size of the KV cache grows linearly with the length of the input sequence, leading to a significant increase in the risk of out-of-memory (OOM) errors. Furthermore, frequent off-chip data transfers cause severe memory bandwidth bottlenecks, severely restricting the system's concurrent processing capabilities and inference efficiency.

[0003] To alleviate the high memory access overhead and GPU memory pressure caused by KV caching, existing technologies have explored various data compression strategies. For example, token-based methods reduce data volume by discarding unimportant tokens, but permanently discarding tokens inevitably leads to the loss of long-tail information; low-bit quantization-based methods are susceptible to interference from feature outliers during the mapping process, facing precision loss or complex inverse quantization overhead; channel pruning-based methods reduce GPU memory usage by removing a subset of less important channels in the feature space, showing great potential.

[0004] In channel pruning techniques, existing schemes can be divided into structured pruning and unstructured pruning. Structured pruning forces the removal of identical channels between grouped terms. While this strict alignment ensures hardware execution efficiency, it sacrifices adaptability to dynamic features. Unstructured pruning, such as the method disclosed in Chinese patent CN121542395A, breaks this limitation, but it has the following limitations: First, its pruning decision is entirely based on the vector norm, ignoring the widespread semantic similarity between KV vectors. This leads to an excessively large cosine distance between the original features and the pruned features under high sparsity, causing a sharp drop in model inference accuracy. Second, existing channel pruning schemes have poor compatibility with low-bit quantization techniques. Large numerical outliers remaining after pruning can introduce catastrophic truncation errors during low-bit mapping, making it difficult to effectively combine the benefits of the two compression methods.

[0005] Furthermore, to improve computational efficiency, some methods attempt to introduce block-level clustering and quantization into the attention module. However, their clustering purpose is limited to dividing the computational task to construct a sparse mask, and the quantization object is still the original vector block, failing to solve the compatibility problem when pruning and quantization are superimposed.

[0006] In summary, existing channel pruning methods suffer from the following technical drawbacks: First, existing unstructured pruning schemes ignore the widespread semantic similarity between key-value vectors. Under high sparsity, the cosine distance between the original features and the pruned features is too large, leading to a sharp decline in model inference accuracy. Second, existing channel pruning schemes have poor compatibility with low-bit quantization techniques. Large numerical outliers remaining after pruning introduce catastrophic truncation errors during low-bit mapping, making it difficult to effectively combine the benefits of both compression methods. Furthermore, existing schemes fail to perform joint optimization based on the semantic distribution characteristics of the data itself, facing the dual dilemma of decreased accuracy and amplified quantization errors when pursuing the ultimate compression rate.

[0007] Therefore, there is an urgent need in this field for a KV cache compression method that can take into account semantic integrity, be compatible with low bit quantization, and achieve dual optimization of video memory and computation. Summary of the Invention

[0008] In view of this, the present invention aims to solve the problem that in the long sequence inference process of existing large language models, the key-value cache grows linearly with the sequence length, which leads to a surge in the risk of memory overflow and causes a serious memory bandwidth bottleneck.

[0009] To achieve the above objectives, this invention provides a key-value caching compression method for large language models based on semantic cluster center offset, including a pre-filling stage and a decoding stage.

[0010] In the pre-filling stage, the input text data sequence is first acquired, and a key matrix and value matrix are generated using a self-attention mechanism. Semantic clustering is performed on the key matrix, dividing the key vectors into multiple semantic clusters, and the cluster center vector for each semantic cluster is extracted. For each key vector, its offset from the cluster center vector of its semantic cluster is calculated, yielding the key offset. Unstructured channel pruning is performed on the key offset, retaining the channels with the largest absolute value in a predetermined proportion, generating a mask matrix and sparse key offsets. The sparse key offsets are then low-bit quantized to obtain quantized sparse key offsets. The quantized sparse key offsets, the mask matrix, and the semantic cluster index of the token are packaged and stored in off-chip memory, while the cluster center vectors are stored in on-chip shared memory.

[0011] During the decoding phase, for the current decoded term, the corresponding quantized sparse key offset, mask matrix, and semantic cluster index are read from off-chip memory. The quantized sparse key offset is dequantized to recover the approximate key offset. Based on the semantic cluster index, the corresponding cluster center vector is read from on-chip shared memory, and the cluster center vector is added to the approximate key offset to reconstruct the approximate key vector. Attention calculation is performed using the approximate key vector.

[0012] Preferably, semantic clustering employs the K-means clustering algorithm, using Euclidean distance as the similarity measure. Alternatively, value vectors can be grouped based on the clustering results of key vectors, or the value matrix can be clustered separately and the cluster center vectors can be extracted.

[0013] Preferably, the preset ratio is an adjustable parameter, dynamically determined based on memory usage, input sequence length, or number of model layers. Low-bit quantization is either INT4 quantization or INT8 quantization.

[0014] Preferably, this application maintains a recently dense window during the decoding stage. When the number of newly generated tokens reaches the window capacity threshold, offset calculation, pruning and quantization are performed on the new KV vectors within the window to generate compressed data packets that are written to off-chip memory.

[0015] Preferably, this application performs the same clustering, offset calculation, pruning, quantization, and storage operations on the value matrix as on the key matrix, and reconstructs the approximate value vector during the decoding stage.

[0016] This application also provides a key-value caching compression system for large language models based on semantic cluster center offset, including: an inference engine, a clustering module, an offset calculation module, a pruning module, a quantization module, a storage management module, and a decompression module. The clustering module performs semantic clustering on the key matrix; the offset calculation module calculates the offset between the key vector and the cluster center vector of its semantic cluster; the pruning module performs unstructured channel pruning on the offsets; the quantization module performs low-bit quantization on the sparse offsets; the storage management module packages and stores the compressed data in off-chip video memory and stores the cluster center vector in on-chip shared memory; and the decompression module reads the compressed data from off-chip video memory and performs inverse quantization and vector reconstruction.

[0017] Preferably, the clustering module employs any one of the following algorithms: K-means clustering, hierarchical clustering, DBSCAN, or Gaussian mixture model.

[0018] This application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: First, this application transforms the original dense feature data with a large numerical distribution range into the difference between the original dense feature data and the cluster center vector through semantic clustering and offset transformation. The offset values ​​are highly concentrated and close to zero. This decentralized operation fundamentally ensures that subsequent compression operations will not destroy the core semantic information and significantly reduces the cosine distance error between the original dense vector and the pruned vector.

[0020] Second, this application applies pruning operations to the offset rather than the original vector, which greatly reduces the overall numerical magnitude of the preserved features and perfectly eliminates the truncation damage caused by large numerical outliers to word-by-word low-bit quantization. It achieves an effective and lossless superposition of the compression benefits of unstructured channel pruning and low-bit quantization, solving the technical problem of incompatibility between pruning and quantization in the prior art.

[0021] Third, this application adopts an asymmetric hierarchical storage architecture, residing the cluster centers in the on-chip shared memory with extremely fast read speeds, and storing the compressed sparse offsets in off-chip video memory. During the decoding stage, the on-chip idle computing power is used to complete the reconstruction, trading computation for video memory bandwidth, which significantly reduces the waiting time for data migration.

[0022] Fourth, this application adopts a streaming processing architecture to maintain a recently dense window during the decoding stage and performs incremental compression on newly generated KV vectors, realizing streaming processing with compression on the same side. This allows the system to use the saved GPU memory space to process larger batch processing tasks at the same time, significantly improving end-to-end inference throughput.

[0023] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall pipeline for reasoning in a large language model integrating the compression method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the KV cache compression process based on semantic cluster center offset in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dynamic decompression and self-attention calculation process of KV cache in an embodiment of the present invention; Figure 4 This is a schematic diagram comparing the end-to-end inference throughput of an embodiment of the present invention with that of an existing method; Figure 5 This is a schematic diagram illustrating the throughput acceleration achieved by combining low-bit quantization in an embodiment of the present invention; Figure 6 This is a schematic diagram comparing the peak video memory usage of the embodiments of the present invention with that of existing solutions; Figure 7 This is a schematic diagram comparing the cosine distance between the compressed features and the original features using different methods in an embodiment of the present invention. Detailed Implementation

[0025] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0026] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0027] Example 1 This embodiment provides a key-value caching compression method for large language models based on semantic cluster center offset. This method transforms the traditional process of directly pruning the original KV vectors into an end-to-end optimized pipeline of semantic clustering, offset transformation, joint compression, and on-chip reconstruction. Before applying Rotary Position Embedding (RoPE), the KV vectors are divided into multiple clusters based on semantic similarity. High-precision cluster centers are extracted and retained, and the original vectors are converted into offsets relative to the cluster centers. This allows subsequent pruning and quantization operations to be performed without compromising semantic integrity, while simultaneously achieving a significant reduction in memory usage and a substantial increase in inference throughput.

[0028] Figure 1 A schematic diagram of the overall pipeline for reasoning in a large language model integrating the compression method of this embodiment is shown. As shown in the figure, the system architecture of this embodiment mainly includes: The inference engine is used to execute attention mechanism algorithms based on the Transformer architecture and generate key-value cache tensors; The KV cache management layer is used to manage the storage and loading of key-value cache tensors, including a compression decision unit, a decompression unit, and a cache allocation unit. The heterogeneous storage layer includes GPU memory and CPU memory. GPU memory is used to allocate a fixed-length buffer pool to store compressed key-value cache data, while CPU memory is used to maintain a long-term cache to save historical key-value cache data.

[0029] The overall inference process includes an offline calibration phase and an online inference phase. The offline calibration phase is optional and is used to determine hyperparameters such as the number of clusters and the pruning threshold. The online inference phase further includes a pre-filling phase and a decoding phase.

[0030] Figure 2 This diagram illustrates the KV cache compression process based on semantic cluster center offset in this embodiment. The following is combined with... Figure 1 Provide a detailed explanation of the operational steps for the pre-filling stage.

[0031] Step S101: Obtain input data and generate KV cache Obtain the text data sequence input by the user during the pre-filling stage of the large language model inference process. Where T is the total number of tokens in the input sequence. The data sequence is processed using a self-attention mechanism to obtain the corresponding key matrix. Sum matrix , where d is the dimension of the hidden layer of the self-attention mechanism preset by the large language model.

[0032] In this embodiment, the key matrix K and value matrix V are obtained before applying Rotated Position Encoding (RoPE). This is because subsequent semantic clustering operations need to be performed in the original semantic space to maintain the semantic similarity of the vectors.

[0033] Step S102: Semantic clustering and cluster center extraction Semantic clustering is performed on the key matrix K to group key vectors with similar semantics into the same cluster. Specifically, this embodiment uses the K-means clustering algorithm, with Euclidean distance as the similarity measure, to divide the T key vectors into M clusters, where M is the preset number of clusters.

[0034] Besides K-means clustering, other clustering algorithms can also be used in this embodiment, such as hierarchical clustering, DBSCAN, and Gaussian mixture model (GMM). All of these clustering algorithms can achieve the goal of clustering semantically similar vectors into a single cluster and extracting cluster centers.

[0035] The number of clusters M can be determined adaptively rather than by a fixed preset. For example, based on the elbow rule: calculate the sum of squared clustering errors (SSE) for different numbers of clusters, and select the inflection point where the rate of decrease in SSE slows down as the number of clusters; or based on the silhouette coefficient: select the number of clusters with the largest average silhouette coefficient; or based on the memory budget: dynamically determine the maximum number of clusters according to the on-chip shared memory capacity to ensure that the cluster centers reside in memory.

[0036] For each cluster Calculate its cluster center vector :

[0037] in, This represents the number of key vectors in the m-th cluster.

[0038] After clustering is completed, all cluster center vectors are... It is stored in high-precision (such as FP16 or FP32) memory on the GPU for high-frequency access during the decoding stage.

[0039] Similarly, the same clustering operation is performed on the value matrix V to obtain the cluster center vectors. It should be noted that the clustering of value vectors can use the same cluster assignment results as the key vectors, that is, the value vectors are grouped according to the clustering results of the key vectors to avoid duplicate calculations.

[0040] In other implementations, the value matrix can be clustered separately, using different numbers of clusters and cluster centers to accommodate the different semantic characteristics of keys and values.

[0041] Step S103: Offset Calculation For each key vector k, determine its cluster. Calculate the offset of the vector from its cluster center. :

[0042] Similarly, for each value vector Calculate the offset :

[0043] Since the cluster centers represent the average semantic features of the cluster, the offset and The numerical range is highly concentrated and close to zero, eliminating large numerical outliers in the original features.

[0044] Step S104: Unstructured channel pruning For offset matrix Perform unstructured channel pruning. Specifically, for each offset vector, calculate the absolute value of each channel, retain only the top r% of channels with the largest absolute values, and set the remaining channels to zero. Here, r is a preset retention ratio, which can be dynamically determined based on memory usage, input sequence length, or number of model layers.

[0045] The pruning retention ratio r can be dynamically adjusted based on various factors. For example, based on the difference in importance between layers: shallow networks retain higher precision, while deep networks are subjected to aggressive pruning; or based on attention scores: for words with higher attention scores, a higher retention ratio is allocated; or based on memory pressure feedback: GPU memory usage is monitored in real time, and when memory usage exceeds a threshold, the retention ratio is reduced.

[0046] In other implementations, offset pruning can be performed globally rather than per word, that is, retaining a preset proportion of elements with the largest absolute value in the entire offset matrix to further improve the compression ratio.

[0047] The pruning operation generates a mask matrix. ,in, The j-th channel representing the offset of the i-th key is retained; otherwise, it is 0. Only the values ​​of the retained channels are kept, resulting in sparse offset data.

[0048] value offset matrix Perform the same pruning operation to obtain the mask matrix. And sparse value offset data.

[0049] Step S105: Low-bit quantization The pruned sparse offsets are then subjected to word-by-word low-bit quantization. Taking INT4 quantization as an example, for the channel value retained in the key offset of the i-th word, its quantization scaling factor is calculated. and zero point :

[0050]

[0051] in, This represents the set of channel indices retained in the i-th lexical unit. The quantized integer representation is:

[0052] Besides INT4, other quantization bit widths can also be used in this embodiment. For example, INT8 quantization is suitable for scenarios with high precision requirements, and INT2 quantization is suitable for scenarios that pursue the ultimate compression rate. In a preferred embodiment, a hybrid precision quantization strategy is adopted: different quantization bit widths are used for different layers and different attention heads.

[0053] Perform the same quantization operation on the value offset.

[0054] Step S106: Asymmetric hierarchical storage quantized sparse offset Mask matrix The cluster IDs are packaged into compressed data blocks and stored in the GPU's off-chip high bandwidth memory (HBM).

[0055] At the same time, the cluster center vector It resides in the GPU's on-chip shared memory in a high-precision format (such as FP16). Since the number of cluster centers M is much smaller than the sequence length T, the on-chip memory overhead is manageable.

[0056] When the number of cluster centers is large and the on-chip shared memory cannot accommodate all cluster centers, the following optimization strategies can be adopted: hierarchical caching, storing frequently accessed cluster centers on-chip and infrequently accessed cluster centers off-chip; compressed storage, performing low-bit quantization on the cluster centers themselves to further reduce storage overhead; register storage, for extreme cases, storing some cluster centers in registers.

[0057] In multi-GPU distributed inference scenarios, a strategy of sharing clustering centers across GPUs can be adopted: synchronizing the clustering centers to the on-chip memory of all GPUs to achieve global uniformity; or a local clustering strategy can be adopted: each GPU independently maintains its local clustering center to reduce communication overhead; or a dynamic load balancing strategy can be adopted: dynamically allocating compression parameters based on the memory usage and computing load of each GPU.

[0058] Figure 3 A schematic diagram of the dynamic decompression and self-attention calculation process of the KV cache in this embodiment is shown. The following is in conjunction with... Figure 2 The steps for generating each new term in the decoding phase are explained in detail.

[0059] Step S201: Read compressed data For the current decoding step, the system reads the compressed data packet of the current term or the term to be processed from the off-chip HBM, including: Quantized sparse bond offset and its mask ; Quantized sparse value offset and its mask ; The cluster index to which this term belongs .

[0060] Step S202: Dequantization and Offset Restoration Perform dequantization on the quantized sparse bond offsets to recover approximate original offsets:

[0061] For a pruned channel (mask is 0), its dequantized value is 0.

[0062] Similarly, the recovery value offset .

[0063] Step S203: Dynamically reconstruct the KV vector Based on clustering index Read the corresponding key clustering center from the on-chip shared memory. Sum value cluster center Adding the dequantized offsets to the cluster centers reconstructs approximate key and value vectors:

[0064]

[0065] This refactoring operation is performed entirely on-chip, without requiring access to off-chip video memory, thus achieving the design goal of trading computation for bandwidth.

[0066] In other implementations, a weighted average of multiple cluster centers can be combined to further improve reconstruction accuracy.

[0067] Step S204: Apply RoPE and perform attention calculation The reconstructed key vector Rotation Position Encoding (RoPE) is applied, and the reconstructed value vector Without applying RoPE, it then participates in the attention calculation of the current query vector q:

[0068] in, These are the reconstructed key matrix and value matrix, respectively.

[0069] The custom GPU kernel operator in this embodiment can be integrated with mainstream attention computing frameworks such as FlashAttention. Specifically, in the block-based computing process of FlashAttention, for each computing block, a dynamic reconstruction operation is performed first, followed by standard attention computing, avoiding intermediate data read / write operations to GPU memory.

[0070] like Figure 3 As shown on the right, this embodiment employs a streaming processing architecture in the decoding stage to support the continuous generation of long sequences.

[0071] During the decoding phase, the system maintains a recently dense window of size W to temporarily store newly generated key-value vectors. When the number of newly generated tokens reaches the window capacity threshold, the following operation is triggered: First, assign the new KV vectors within the window to existing semantic clusters, or trigger incremental clustering updates.

[0072] Cluster centers can be updated periodically, for example, every N decoding steps, incrementally updating the cluster centers based on the newly generated KV vectors to adapt to changes in semantic distribution. Incremental updates can be implemented using methods such as moving averages or online K-means.

[0073] Then, offset calculation, pruning, and quantization are performed on the vectors within the new window to generate compressed data packets.

[0074] Next, the newly generated compressed data packet is written to the off-chip HBM, and the metadata is updated.

[0075] Finally, based on the memory budget, historical compressed data is selectively discarded to achieve a dynamic balance in memory usage.

[0076] The window size W can be dynamically adjusted based on the length of the input sequence, or adaptively set according to the distribution of attention scores.

[0077] The compression method in this embodiment can be combined with the PagedAttention mechanism of inference frameworks such as vLLM. The compressed KV cache data is stored in a paged format, and the physical address of the compressed data block is managed through a page table to achieve efficient memory allocation and sharing.

[0078] This embodiment was experimentally validated on the LLaMA 3.1-8B model, using long-text benchmark datasets such as LongBench and ZeroSCROLLS. The experimental configuration is as follows: Hardware environment: The main processor uses a 14-core vCPU (Intel Xeon Gold 6348 @ 2.60GHz) as the scheduling node, equipped with a single A800-80GB computing card.

[0079] Software environment: The operating system is based on Linux (Ubuntu 22.04), the underlying layer calls the CUDA 12.8 parallel computing platform, and the upper-layer algorithm logic is built on Python 3.12 and PyTorch 2.8.0 deep learning framework.

[0080] Baseline methods: Dense baseline (uncompressed), Mustafar, SparK, SnapKV.

[0081] Table 1 shows the average accuracy comparison of different pruning methods on the LongBench benchmark. As can be seen from Table 1, when the sparsity reaches a very high level, the accuracy of existing methods (Mustafar, SparK) drops sharply, while the method in this embodiment still maintains an overall score comparable to the dense baseline.

[0082] Table 1

[0083] Figure 4 and Figure 5 The comparison between the method of this embodiment and existing methods in terms of end-to-end throughput is shown. Experiments demonstrate that when concurrency scales up and the system bottleneck shifts to memory bandwidth, the method of this embodiment significantly overcomes memory access limitations. In particular, when combined with low-bit quantization, the end-to-end throughput can reach more than three times that of the dense benchmark.

[0084] Figure 6 The peak memory usage of this embodiment is compared with that of existing solutions. Traditional dense benchmarks face a serious risk of memory overflow as batch size increases, while the method in this embodiment exhibits an extremely smooth memory growth curve. By combining low-bit quantization, peak memory consumption is reduced by approximately 50%, and the concurrent batch size that the system can support increases to three times that of dense benchmarks.

[0085] Figure 7 The cosine distance between the compressed features and the original features is shown in the diagram. The cosine distance error of the method in this embodiment is extremely low, significantly better than existing pruning schemes.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A key-value caching compression method for large language models based on semantic cluster center offset, characterized in that, include: Pre-filling stage: The input text data sequence is acquired, and a key matrix and a value matrix are generated using a self-attention mechanism. Semantic clustering is performed on the key matrix to divide the key vectors into multiple semantic clusters, and the cluster center vector of each semantic cluster is extracted. For each key vector, the offset between it and the cluster center vector of its semantic cluster is calculated to obtain the key offset. Unstructured channel pruning is performed on the key offset to retain the channels with the largest absolute value in a preset proportion, generating a mask matrix and sparse key offsets. The sparse bond offset is quantized using low-bit quantization to obtain the quantized sparse bond offset. The quantized sparse key offset, the mask matrix, and the semantic cluster index to which the word belongs are packaged and stored in off-chip video memory, while the cluster center vector is stored in on-chip shared memory. Decoding phase: For the current decoded term, the corresponding quantized sparse key offset, mask matrix, and semantic cluster index are read from off-chip memory; the quantized sparse key offset is dequantized to recover the approximate key offset; according to the semantic cluster index, the corresponding cluster center vector is read from on-chip shared memory, and the cluster center vector is added to the approximate key offset to reconstruct the approximate key vector; attention calculation is performed using the approximate key vector.

2. The method according to claim 1, characterized in that, The semantic clustering uses the K-means clustering algorithm, with Euclidean distance as the similarity measure.

3. The method according to claim 1, characterized in that, The semantic clustering includes: grouping value vectors according to the clustering results of key vectors, or clustering value matrices separately and extracting value cluster center vectors.

4. The method according to claim 1, characterized in that, The preset ratio is an adjustable parameter that is dynamically determined based on memory usage, input sequence length, or number of model layers.

5. The method according to claim 1, characterized in that, The low-bit quantization is either INT4 quantization or INT8 quantization.

6. The method according to claim 1, characterized in that, The method further includes: maintaining a recently dense window during the decoding stage; when the number of newly generated tokens reaches the window capacity threshold, performing offset calculation, pruning, and quantization on the new KV vectors within the window, and generating a compressed data packet to be written to off-chip memory.

7. The method according to claim 1, characterized in that, The method further includes performing the same clustering, offset calculation, pruning, quantization, and storage operations on the value matrix as on the key matrix, and reconstructing the approximate value vector during the decoding stage.

8. A key-value caching and compression system for large language models based on semantic cluster center offset, characterized in that, include: The inference engine is used to execute the attention mechanism algorithm and generate key-value cache tensors; The clustering module is used to perform semantic clustering on the key matrix, dividing the key vectors into multiple semantic clusters and extracting the cluster center vector of each semantic cluster; The offset calculation module is used to calculate the offset between the key vector and the cluster center vector of its semantic cluster. The pruning module is used to perform unstructured channel pruning on the offset to generate a mask matrix and sparse offsets. A quantization module is used to perform low-bit quantization on the sparse offset; The storage management module is used to package and store the quantized sparse offsets, mask matrices and semantic cluster indexes in off-chip video memory, and store the cluster center vectors in on-chip shared memory. The decompression module is used to read compressed data from off-chip video memory, perform dequantization and vector reconstruction, and recover the approximate key vector; The system is used to perform the method according to any one of claims 1 to 7.

9. The system according to claim 8, characterized in that, The clustering module employs any one of the following algorithms: K-means clustering, hierarchical clustering, DBSCAN, or Gaussian mixture model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images