Large language model-oriented data compression method, reasoning system and related equipment
By decomposing and hybridizing the key-value cache tensor of a large-scale language model into a sparse exponential high-order plane and a dense mantissa low-order plane, the problems of high memory consumption and high latency of KV Cache are solved, achieving efficient lossless compression and improved throughput.
Patent Information
- Application Number
- CN202511691700.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-02-10
AI Technical Summary
Existing key-value caches for large-scale language models consume a lot of GPU memory, have high compression latency, and insufficient throughput, making them difficult to deploy efficiently in production environments.
An exponential sparsity and hybrid vectorization compression method is adopted to decompose the key-value cache tensor into a sparse exponential high-order plane and a dense tail low-order plane. A compressed data matrix is generated by hybrid vectorization compression, and then stored and decompressed in combination with GPU parallel structure.
It achieves efficient and lossless compression of key-value cache tensors, significantly reducing GPU memory usage and data transfer overhead, improving inference throughput and reducing latency, and is suitable for large model inference scenarios.
Smart Images
Figure CN121508546A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data compression method, inference system and related equipment for large language models. Background Technology
[0002] With the widespread application of large-scale language models in tasks such as natural language processing, code generation, and multimodal reasoning, the computational and storage overhead of their reasoning process has increased dramatically. Especially in autoregressive generation, to support contextual dependencies and long sequence inputs, the model needs to maintain a key-value cache (KV Cache) in each attention mechanism layer. This cache stores the feature representations of historical tokens, and its space consumption grows linearly with the context length, becoming one of the main bottlenecks affecting inference performance and cost.
[0003] Currently, to reduce the memory footprint of KV cache, academia and industry mainly employ two types of methods: lossy compression and lossless compression. The core idea of lossy compression is to map floating-point values to low-bit discrete values using quantization techniques, thereby reducing the storage width of a single element. Lossless compression includes two typical techniques: entropy-based coding and literal coding. The former, such as Huffman coding and ANS coding, achieves compression by assigning shorter codes to high-frequency data; the latter directly utilizes the repetition in the data bit patterns for compression.
[0004] However, existing solutions have many problems. For example, for lossy compression algorithms, quantization errors accumulate gradually over multiple rounds of autoregressive generation, leading to semantic drift and accuracy degradation. Their "near-lossless" experimental results often depend on specific benchmark tasks, while their performance is unstable in real-world applications (such as multi-turn dialogues and long context generation). Therefore, low-bit formats lack general support in mainstream inference engines (such as vLLM and SGLang), making them difficult to deploy in production environments. For lossless compression algorithms, although these methods can strictly maintain numerical accuracy, they have significant shortcomings in practical applications. On the one hand, entropy coding methods have strong order dependencies, making parallelization difficult and resulting in extremely low compression and decompression throughput in high-performance GPU environments. On the other hand, literal compression methods cannot effectively utilize the redundancy in modern floating-point formats (such as BFloat16) because they use "offset exponent" representation, causing floating-point numbers with similar values to present completely different binary modes at the bit level, making it difficult for the compressor to discover compressible modes. Summary of the Invention
[0005] This application provides a data compression method, inference system, and related equipment for large language models, which can significantly improve throughput and reduce latency during compression or decompression.
[0006] The first aspect of this application provides a data compression method for large language models, including:
[0007] The input key-value cache tensor is subjected to exponential sparsification to generate a sparse exponential high-order plane and a dense mantissa low-order plane; the sparse exponential high-order plane is used to characterize the sparse tensor of the key-value cache tensor, and the dense mantissa low-order plane is used to characterize the dense tensor of the key-value cache tensor.
[0008] The sparse tensor is subjected to hybrid vectorization compression to generate a compressed data matrix; the data matrix includes a column index matrix and a value matrix.
[0009] The dense tensor, the column index matrix, and the value matrix are stored according to a preset tensor data structure, and decompressed when needed to restore the data corresponding to the original key-value cache tensor.
[0010] A second aspect of this application provides a data compression system for large language models, comprising:
[0011] The processing unit is used to perform exponential sparsification processing on the input key-value cache tensor to generate a sparse exponential high-order plane and a dense mantissa low-order plane; the sparse exponential high-order plane is used to characterize the sparse tensor of the key-value cache tensor, and the dense mantissa low-order plane is used to characterize the dense tensor of the key-value cache tensor.
[0012] A generation unit is used to perform hybrid vectorization compression on the sparse tensor to generate a compressed data matrix; the data matrix includes a column index matrix and a value matrix.
[0013] The storage unit is used to store the dense tensor, the column index matrix, and the value matrix according to a preset tensor data structure, and to decompress them when needed to restore the data corresponding to the original key-value cache tensor.
[0014] The data compression system for large language models provided in the second aspect of this application is used to execute the data compression method for large language models described in the first aspect.
[0015] A third aspect of this application provides a data compression system for large language models, comprising:
[0016] The inference engine is used to execute the attention mechanism Transformer algorithm and generate key-value cache tensors;
[0017] The key-value cache (KV) management layer is used to manage the storage and loading of the key-value cache tensors;
[0018] The heterogeneous storage layer includes GPU memory and CPU memory; the GPU memory is used to allocate a fixed-length buffer pool to store the data corresponding to the key-value cache tensor; the CPU memory is used to maintain a long-term cache to save the data corresponding to historical key-value cache tensors.
[0019] The data compression system for large language models provided in the third aspect of this application is used to execute the data compression method for large language models described in the first aspect, and is used to perform compression or decompression operations when the key-value cache tensor is stored or loaded.
[0020] A fourth aspect of this application provides a data compression apparatus for large language models, comprising:
[0021] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;
[0022] The memory is either a short-term storage memory or a persistent storage memory;
[0023] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the data compression method for large language models as described in the first aspect.
[0024] A fifth aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the data compression method for large language models described in the first aspect.
[0025] A sixth aspect of this application provides a computer program product including instructions that, when executed on a computer, cause the computer to perform the data compression method for large language models described in the first aspect.
[0026] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The data compression method for large language models disclosed in this application achieves efficient and lossless compression of key-value cache tensors through a collaborative design of exponential sparsity and hybrid vectorization compression. Simultaneously, since the compressed data can be stored in a contiguous memory layout, combined with single-kernel fusion computing, it significantly reduces GPU memory usage and data transfer overhead, thereby simultaneously improving throughput and reducing latency in large model inference scenarios. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0028] Figure 1 This is a flowchart illustrating a data compression method for large language models disclosed in an embodiment of this application;
[0029] Figure 2 This is a flowchart illustrating another data compression method for large language models disclosed in an embodiment of this application;
[0030] Figure 3 This is a flowchart illustrating another data compression method for large language models disclosed in an embodiment of this application;
[0031] Figure 4 This is a schematic diagram of the architecture of a reasoning system for large language models disclosed in an embodiment of this application;
[0032] Figure 5 This is a flowchart illustrating an exponential sparsity step disclosed in an embodiment of this application;
[0033] Figure 6 This is a flowchart illustrating a hybrid vectorization step disclosed in an embodiment of this application;
[0034] Figure 7 This is a schematic diagram of the structure of a data compression system for large language models disclosed in an embodiment of this application;
[0035] Figure 8 This is a schematic diagram of the structure of a data compression device for large language models disclosed in an embodiment of this application. Detailed Implementation
[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0037] To address the issues of significant KV cache usage, high compression latency, and insufficient throughput in the large language model inference process described in the background, this application discloses a data compression method for large language models.
[0038] The application primarily discloses a fine-grained KV cache lossless compression method for large language model inference. Specifically, this example designs a GPU-friendly compression algorithm based on Zigzag Bitplane Exponent Sparsification (ZBES) technology. This algorithm comprehensively utilizes the characteristics of floating-point exponent distribution and the parallel structure features of the GPU to achieve high compression ratio and high throughput while ensuring complete numerical loss. The technical solution of this application mainly includes two core modules:
[0039] (1) Exponential sparsity module: In view of the exponential offset pattern of BFloat16 floating-point format, Zigzag-style exponential remapping and bit layer decomposition are adopted to generate a sparse bit plane to enhance compressibility.
[0040] (2) Hybrid vectorization module: For the sparse matrix structure after exponential sparsification, a block-level vectorization compression format with the backbone path and overflow path working together is proposed to balance compression ratio and decoding parallelism.
[0041] The two modules described above are integrated during implementation and executed within the CUDA (Compute Unified Device Architecture) kernel, enabling unified data flow processing and avoiding the overhead of multi-core calls and memory round trips, thereby significantly improving compression and decompression throughput. The following sections will describe the working principles, data structures, and operational relationships of these two modules.
[0042] Please refer to the following: Figure 1 , Figure 1 This is a flowchart illustrating a data compression method for large language models disclosed in an embodiment of this application. It includes steps 101-103.
[0043] 101. Perform exponential sparsification on the input key-value cache tensor to generate a sparse exponential high-order plane and a dense mantissa low-order plane.
[0044] It should be noted that step 101 in this embodiment is mainly implemented by the exponential sparsity module. The purpose of the exponential sparsity module is to structure the redundant information of the exponential part in the floating-point cache tensor to form a sparse bit plane that can be compressed in parallel.
[0045] In one specific embodiment, the key-value cache (KV Cache) tensor input during model inference is exponentially sparsified using an exponential sparsification module, thereby generating a sparse exponential high-order plane and a dense mantissa low-order plane. The sparse exponential high-order plane is used to characterize the sparse tensor of the key-value cache tensor. The dense mantissa low-order plane is used to characterize the dense tensor of the key-value cache tensor. .
[0046] In some embodiments, the exponent sparsification module, targeting the exponent offset pattern of the BFloat16 floating-point format (including the sign part, exponent part, and mantissa part), splits and reassembles the key-value cache tensor into an incompressible part (including the sign part and mantissa part) and an exponent part. Then, by performing a Zigzag-style exponent remapping on the exponent part, the literal value of the exponent part is mapped to a positive value, thereby increasing the concentration of the exponent distribution. Next, the mapped exponent is divided into dense bits (low-order bits) and sparse bits (high-order bits). The dense bits and the incompressible part are combined to form a dense tensor. By preserving the sparse bits and combining different key-value cache tensors, a sparse tensor is obtained. .
[0047] In some embodiments, the processing flow of the exponential sparsity module mainly includes (1) input preprocessing and block layout, (2) exponent and mantissa decomposition, (3) Zigzag-style exponential remapping, (4) bit layer decomposition and structured sparsity, and (5) sparse plane packing and storage. The above steps will be implemented in [the following steps]. Figure 2 The illustrated embodiments are described in detail below.
[0048] 102. Perform hybrid vectorization compression on the sparse tensor to generate a compressed data matrix.
[0049] It should be noted that step 102 in this embodiment is mainly implemented by the hybrid vectorization module, which is primarily designed for the sparse tensors generated by the exponential sparsification module. By leveraging its structured sparse features for vectorized encoding and storage optimization, a balance between compression ratio and decoding parallelism is achieved. For easier understanding, please refer to [link to relevant documentation]. Figure 3 The illustrated embodiment.
[0050] In one specific embodiment, after processing different key-value cache tensors in parallel through matrix sparsification, a sparse tensor is obtained. Then sparse tensors Through parallel processing, the data is scanned according to its linear arrangement, extracting the non-zero values and their column indices for each row, thereby generating two intermediate matrices: a column index matrix and a value matrix (i.e., the data matrix mentioned above). The number of rows in the column index matrix and the value matrix corresponds to the number of rows in its linear arrangement, while the number of columns depends on the number of non-zero elements in each row.
[0051] In some embodiments, each sparse tensor Each row of non-zero elements is used as a counting standard. By counting the number of non-zero elements in each row, the columns are combined into a column index matrix (marking the number of non-zero elements in each row) and a value matrix (marking the combined values recorded in the column index matrix).
[0052] 103. Store the dense tensor, column index matrix, and value matrix according to the preset tensor data structure, and decompress them when needed to restore the data corresponding to the original key-value cache tensor.
[0053] Then, the data in the fixed-width (by column) columns of the column index matrix and value matrix are divided into main paths. and overflow path, the main path With dense tensor Combine them to obtain the main path data ( + The overflow paths are combined to obtain the overflow path data. Then the main path data ( + and overflow path data Compress and store the data.
[0054] Furthermore, when decompression is required, the trunk path data ( + The data is decoded in parallel by the GPU, with each thread corresponding to one compressed block line, restoring non-zero elements according to a fixed structure. Overflow path data. The missing data is then filled in using a metadata index. The decoded sparse tensor With dense tensor By merging and reversing the exponential remapping and floating-point assembly, the original BFloat16 value can be fully restored.
[0055] This embodiment discloses a data compression method for large language models. Through the collaborative design of exponential sparsity and hybrid vectorization compression, it achieves efficient and lossless compression of key-value cache tensors. Simultaneously, since the compressed data can be stored in a contiguous memory layout, combined with single-kernel fusion computation, it significantly reduces GPU memory usage and data transfer overhead, thereby simultaneously improving throughput and reducing latency in large model inference scenarios.
[0056] Figure 2 This is a flowchart illustrating another data compression method for large language models disclosed in an embodiment of this application. It includes steps 201-204.
[0057] 201. Divide the key-value cache tensor into multiple compressed blocks according to the set dimensions of the large language model, and decompose the floating-point elements in each compressed block into exponent and mantissa to obtain the corresponding floating-point exponent part, floating-point mantissa part and floating-point sign part.
[0058] In this embodiment, in conjunction with step 101 above, the exponential sparsity module mainly consists of 5 steps. For ease of understanding, these steps can be combined with... Figure 5 , Figure 5 This is a flowchart illustrating an exponential sparsity step disclosed in an embodiment of this application.
[0059] In one specific embodiment, the KV Cache tensor in model inference (typically of shape ...) ) Tokens are divided into shapes according to the set dimensions. The three-dimensional layout, in which For sequence length, To set the number of tokens for a given dimension, The number of compressed blocks per token, with each block containing 512 BFloat16 elements, corresponds to the GPU's thread block size. The BFloat16 floating-point format consists of a sign part (1 bit), an exponent part (8 bits), and a mantissa part (7 bits). This layout ensures that each thread block processes a complete compressed block (which can also be understood as consisting of several floating-point numbers), thereby achieving thread-level alignment and memory access continuity.
[0060] Furthermore, the BFloat16 elements are split and recombined to form an incompressible part (including the sign part and the mantissa part) and an exponent part.
[0061] In some embodiments, combined with Figure 5 As shown, each BFloat16 element in the input tensor is decomposed bitwise, dividing its 16-bit binary representation into a 1-bit sign bit, an 8-bit exponent part, and a 7-bit mantissa part. The distribution of the exponent part is usually concentrated near the bias value (e.g., around 127), meaning that most exponent values are close to zero offset. In a certain BFloat16 element, its floating-point element is 1111111011101010, where the first sign bit is "1", the middle exponent part is "111111011", and the mantissa part is "101010". The incompressible part after decomposition is the combination of the sign and mantissa parts, "11101010", and the corresponding exponent part is still "111111011".
[0062] 202. Perform Zigzag-style remapping on the decomposed floating-point exponent part, and decompose the remapped exponent part into high-order sparse bits and low-order dense bits.
[0063] Then, the decomposed floating-point exponent part is subjected to Zigzag-style exponent remapping, thereby decomposing the remapped exponent part into high-order sparse bits and low-order dense bits. It can be understood that Zigzag-style remapping is a specific data reading or padding order that scans a matrix (usually two-dimensional) back and forth in a zigzag pattern, with the aim of maintaining the adjacency of adjacent data in two-dimensional space.
[0064] In one specific embodiment, firstly, any compressed block is searched to determine the range of the floating-point element set within that compressed block, obtaining the starting value and ending value of the range; wherein, the ending value of the range is obtained from the starting value and a preset floating-point parameter; the range is... , The starting value of the interval. The floating-point parameters are preset (a user-defined hyperparameter). The floating-point exponent is then symmetrically compressed to the lower-order interval using a remapping function, so that the higher-order sparse bits present a continuous zero region; where the remapping function is... , It is the literal value of the corresponding exponent part in the floating-point exponent part.
[0065] In some embodiments, the positive or negative exponent portion of the floating-point exponent portion can be symmetrically compressed to the lower bit range using a remapping function, thereby making the higher sparse bits present a continuous zero region.
[0066] Wherein, the remapping function is , This is the literal value of the exponent portion of the floating-point exponent part. In other words, This refers to the literal value of the exponent part in the BFloat16 format (according to its encoding format, the actual value represented by the exponent part is the literal value + 127).
[0067] In some embodiments, this mapping symmetrically compresses positive and negative exponents (positive and negative exponents refer to the exponent portion in the BFloat16 format. This means that after mapping, both positive and negative exponents will be mapped to a positive value) into the lower bit range, significantly improving the concentration of the exponent distribution and causing the higher bits to present a large number of consecutive "0" regions. Experiments show that after this transformation, more than 95% of the exponent values have the highest 4 bits as 0.
[0068] The mapped 8-bit exponent is divided into a high 4 bits and a low 4 bits. The high 4 bits form a sparse bit plane, storing the high-order part. The subsequent low 4 bits are concatenated with the original 7-bit mantissa and the sign bit to form a dense bit plane. Understandably, after mapping, the literal value will be smaller, so it tends to occupy several bits on the right (possibly less than 4 bits). This is the effect after mapping. For example, combined with... Figure 5 As shown, the exponent part "111111011" is remapped in a Zigzag style to obtain "00000100". The exponent after the above mapping is then split into dense bits (lower 4 bits) "0100" and sparse bits (higher 4 bits) "0000".
[0069] 203. Concatenate the high-order sparse bits of different compressed blocks to generate a sparse exponential high-order plane.
[0070] Then, by splicing together the high-order sparse bits of different compressed blocks, a sparse exponential high-order plane can be generated.
[0071] In one specific embodiment, combined with Figure 5 As shown, the high-order sparse bits of different compressed blocks (or understood as different key-value cache tensors) are concatenated, that is, different elements (element 1, element 2, etc.) are concatenated and combined to generate a sparse exponential high-order plane. At this point, the corresponding element 1 or element 2 should be 4 bits. For example, by combining the sparse bits "0000" of element 1 and the sparse bits of element 2, a sparse tensor can be obtained. .
[0072] 204. Concatenate the low-order dense bits, floating-point sign part, and floating-point mantissa part of the same compressed block to obtain the low-order dense plane of any compressed block; combine the low-order dense planes of different compressed blocks to obtain the dense mantissa low-order plane.
[0073] Simultaneously with step 203, the low-order dense bits, floating-point sign portion, and floating-point mantissa portion of the same compressed block can be concatenated to obtain the low-order dense plane corresponding to any compressed block. Then, the low-order dense planes of different compressed blocks are combined to obtain the dense mantissa low-order plane.
[0074] In one specific embodiment, combined with Figure 5 As shown, the lower 4 dense bits "0100" are concatenated with the original mantissa "1101010" and the sign part "1" to form the dense plane "11101010" for element 1. At this point, element 1 is 12 bits. Then, different elements (element 1, element 2, etc.) are combined and concatenated to obtain the dense mantissa lower bit plane, i.e., the dense tensor. .
[0075] Therefore, for the entire key-value cache tensor, the original 16-bit floating-point number is re-represented as a pair of tensors. . This represents the high-order plane of the sparsity index. This represents the low-order plane of dense mantissas.
[0076] In some embodiments, sparse tensors are obtained Then, for sparse tensors Bit packing is performed, merging two 4-bit values into one byte, that is, extracting the rightmost four bits of the mapped result. These extracted four bits are then paired and concatenated into a single byte. (The dense tensor...) Stored in-situ to support fast access. This allows sparse tensors to be stored... It accounts for only 25% of the original data, and most of the bits are 0, which lays the foundation for efficient compression of the subsequent hybrid vectorization module.
[0077] This embodiment discloses a data compression method for large language models, achieving efficient compression and storage optimization for large-scale language model key-value caches (KV Cache). Specifically, by decomposing BFloat16 format floating-point numbers into sign, exponent, and mantissa, and further performing Zigzag-style remapping and splitting on the exponent, the original 16-bit data is reconstructed into a sparse exponent high-bit plane and a dense mantissa low-bit plane. In the sparse plane, the high-bit values exhibit a large number of consecutive zero values due to the concentrated mapping, resulting in the sparse tensor accounting for only 25% of the original data. Simultaneously, bit packing technology is supported to merge two 4-bit values into one byte, further compressing storage space. This structured compression strategy not only reduces memory usage but also reduces bandwidth pressure during data transmission, making it particularly suitable for memory-constrained inference scenarios. Secondly, this scheme balances data access locality and computational efficiency during compression. By dividing the KV Cache tensor into compression blocks aligned with GPU thread blocks (e.g., 512 BFloat16 elements) according to the token dimension, each thread block can independently process a complete compression block, achieving thread-level parallel optimization and memory access continuity. The dense mantissa plane is stored in its original position, ensuring that the frequently accessed mantissa and sign parts can be read quickly. The efficient compression of the sparse plane also lays the foundation for subsequent hybrid vectorization processing, thus achieving a good balance between compression ratio and inference speed. Furthermore, the introduction of Zigzag remapping is key to improving compression performance. This mapping symmetrically compresses positive and negative exponents to the lower bit range, significantly enhancing the concentrated distribution characteristics of exponent values. Experiments show that over 95% of the exponent values are zero in the higher 4 bits, making the sparse plane highly compressible. This characteristic not only benefits the application of traditional compression algorithms but also facilitates the encoding, transmission, and subsequent computation of sparse tensors, making it particularly suitable for hardware acceleration and optimization of dedicated instruction sets. In addition, this method has strong versatility and scalability. Although the example uses BFloat16 format and GPU thread block size, its ideas of block division, decomposition, remapping, and plane reconstruction can be adapted to other floating-point formats (such as FP16 and FP32) and different hardware architectures. In scenarios that require processing massive KV caches, such as supporting long sequence generation and multimodal large models, this technology can effectively alleviate storage bottlenecks and support model inference with larger batches or longer contexts.
[0078] Figure 3 This is a flowchart illustrating another data compression method for large language models disclosed in an embodiment of this application. It includes steps 301-306.
[0079] 301. Scan according to the row arrangement of the sparse tensor, and extract the non-zero values of each row and the column index of the corresponding non-zero values of each row.
[0080] In this embodiment, steps 102 and 103 described above are primarily executed by the hybrid vectorization module. For ease of understanding, it can be combined with... Figure 6 , Figure 6 This is a flowchart illustrating a hybrid vectorization step disclosed in an embodiment of this application.
[0081] In one specific embodiment, a key-value cache tensor (KV cache) in accordance with After the matrix (Bfloat16 format) is divided into different compressed blocks, it needs to be sparsified and processed in parallel (as described above). Figure 2 As shown), thus obtaining the sparse tensor. (4-bit width) and dense tensors (12-bit width). Then, for sparse tensors... In contrast, parallel processing is required.
[0082] Specifically, first consider the sparse tensor The system scans row by row (i.e., the linear arrangement within the compressed block) to extract the non-zero values and their column indices for each row. The column indices are used to characterize the position of the non-zero values in each row of the sparse tensor.
[0083] 302. Process the non-zero values or column indices of each row in parallel to obtain the column index matrix and the value matrix.
[0084] Therefore, when processing the non-zero values or column indices of each row separately in parallel, two intermediate matrices can be generated: a column index matrix and a value matrix. The number of rows in each matrix corresponds to the number of rows in the compressed block, and the number of columns depends on the number of non-zero elements in the row. In other words, the number of rows in each matrix corresponds to the number of rows in the sparse tensor, and the number of columns in each matrix is related to the number of non-zero values in each row. The value matrix is used to record the corresponding non-zero values in the sparse tensor.
[0085] Among them, combined Figure 6 The sparse tensor shown sparse tensor Most positions are 0 (white), with only a few positions having values. Each value is... Figure 5 The leftmost four bits of the mapped exponent. These four bits are mostly 0, so combining them forms a sparse tensor. For example, combining Figure 6 As shown, in a certain sparse tensor In the column index matrix, the first row has one non-zero element (located at position 1, defined as 0, 1, 2, 3, 4 in any row), the second row has three non-zero elements (located at positions 0, 2, and 4), the third row has no non-zero elements, and the fourth row has two non-zero elements (located at positions 3 and 4). Therefore, in the column index matrix, the corresponding first row is 1, the second row is 0, 2, 4, and the fourth row is 3, 4. Alternatively, it can be understood that in the column index matrix, the first row has only one non-zero value (blue), so the first row of the column index matrix records only the number 1 (the second item counting from 0). Meanwhile, the second row has three non-zero values, the 1st, 3rd, and 5th. Therefore, the column index matrix of the second row records 0, 2, 4 (the 1st, 3rd, and 5th items counting from 0). Similarly, the encoding rules for the value matrix are consistent with those for the column index matrix. However, unlike the column index matrix which records serial numbers, the value matrix records... The corresponding non-zero value in the middle.
[0086] In some embodiments, due to dense tensors It has non-zero elements, so it is directly represented by gray.
[0087] 303. A dual-path encoding structure is adopted to store dense tensors, column index matrices, and value matrices.
[0088] For the column index matrix and value matrix, a dual-path encoding structure can be used to store the dense tensor, column index matrix, and value matrix. This dual-path encoding structure includes a backbone path and an overflow path. The backbone path uses a fixed-column-width three-dimensional tensor structure to store the column indices of non-zero values in the column index matrix and the corresponding column index values in the value matrix. The overflow path stores non-zero values in the column index matrix and value matrix that exceed the fixed column width. The overflow path includes row-level metadata, overflow column indices, and overflow values; the row-level metadata records the number of overflows per row.
[0089] In one specific embodiment, combined with Figure 6 As shown, a dual-path design is adopted. The main path is set with a fixed column width for the vast majority of sparse rows (few and similar non-zero rows). It employs a compact three-dimensional tensor structure for storage, where each element contains a pair (column_index, value). This fixed-width structure supports memory-aligned access and warp-level parallel decoding. Fixed column width. This is used to specify how many columns on the left side of the column index matrix and value index matrix will be selected and incorporated into the main path data.
[0090] Secondly, for cases where the number of non-zero elements in a few rows exceeds the limit, an additional overflow buffer is set up, which includes (1) a row-level metadata array to record the overflow quantity of each row; (2) an overflow column index array (merged column data); and (3) an overflow value array (merged value data).
[0091] 304. Merge the data in the main path with the dense tensor to obtain the main path data; combine the data in the overflow path to obtain the overflow path data.
[0092] In some embodiments, the main path data extracted in step 303 With dense tensor The data is then merged to obtain the main path data. + The data in the overflow path is combined to obtain the overflow path data. .
[0093] 305. Compress and store the main path data and overflow path data.
[0094] In some embodiments, the main path data ( + and overflow path data Compressed storage is performed. However, this embodiment does not limit the specific method of compressed storage.
[0095] 306. By decoding the backbone path data in parallel and combining it with the overflow path data, the sparse exponential high-order plane can be recovered.
[0096] In some embodiments, during decoding, the sparse exponential high-order plane can be recovered by decoding the backbone path data in parallel and combining it with the overflow path data. Specifically, during decoding, the offset is calculated using prefix sums to achieve rapid recombination of overflow elements. This structure ensures that the backbone path maintains a regular shape while also accommodating long-tail data distribution. During decompression, the backbone path is decoded in parallel by the GPU: each thread corresponds to one compressed block line, restoring non-zero elements according to a fixed structure; the overflow path is padded using metadata indexing. The decoded data... and By merging and reversing the exponential remapping and floating-point assembly, the original BFloat16 value can be fully restored.
[0097] This embodiment discloses a data compression method for large language models. The hybrid vectorization and dual-path encoding scheme, through a highly structured sparse data processing flow, significantly improves decoding parallelism and storage scalability while maintaining high compression efficiency. First, the scheme transforms sparse tensors into structured column index matrices and value matrices through row and column scanning and non-zero element extraction, laying the foundation for subsequent efficient encoding. The high-order exponents in the sparse tensor, after Zigzag remapping, exhibit high sparsity (over 95% of the bits are zero), drastically reducing the number of non-zero elements. By extracting the non-zero values and column indices of each row, the system transforms irregular sparse data into a regularized intermediate representation, preserving the semantic structure of the original data and creating conditions for parallel processing. This transformation significantly reduces the complexity of data processing, making it particularly suitable for parallel computing architectures such as GPUs. Second, the dual-path encoding structure innovatively balances the needs of regular storage and flexible expansion, effectively addressing the long-tail distribution problem in sparse data. The backbone path uses a fixed-column-width three-dimensional tensor to store the non-zero elements of most rows, ensuring memory access alignment and thread-beam-level parallel decoding efficiency. This design enables mainstream sparse rows to be stored in a compact format and supports high-speed random access. For a few rows with a large number of non-zero elements, the overflow path is managed through three components: row-level metadata, overflow column index, and overflow value. This avoids storage waste caused by expanding fixed column widths due to individual abnormal rows and ensures that all data is preserved intact. Experiments show that this structure maintains over 95% of the data in the backbone path while transparently handling abnormally sparse rows. Furthermore, this encoding scheme achieves a balance between compression ratio and decoding speed. By fusing dense tensors with backbone path data, the system manages the frequently accessed mantissa portion and the sparse exponent backbone path in a unified manner, reducing the randomness of data access. The independent storage of the overflow path ensures that the decoding process is not degraded by a small amount of abnormal data. In the compression storage stage, the regular structure of the backbone path facilitates the application of traditional compression algorithms to further reduce storage footprint, while the compact encoding of the overflow path minimizes additional overhead. Finally, the parallel decoding mechanism of this scheme greatly improves decompression efficiency. During the inference phase, the system efficiently recovers the sparse exponential high-order plane by decoding the backbone path data in parallel at the GPU thread level and quickly locating overflow elements using prefix sum calculations. This design ensures that the decompression process is almost never a bottleneck in the inference pipeline, performing exceptionally well in real-time inference and large-batch processing scenarios. Furthermore, the combination of the dual-path structure and hybrid vectorization provides a unified and efficient processing framework for data with different sparsity patterns, exhibiting good versatility and scalability.
[0098] The following is combined with Figures 1 to 3 The illustrated embodiment describes the overall operation flow of this algorithm integrated into the inference deployment system. See also... Figure 4 , Figure 4This is a schematic diagram of the architecture of a reasoning system for large language models disclosed in an embodiment of this application.
[0099] like Figure 4 As shown, the inference system mainly consists of three parts: an inference engine layer, a KV Cache management layer, and a heterogeneous storage layer. These layers collaborate through a unified interface to achieve efficient cross-request reuse and KV Cache compressed access mechanisms. First, the inference engine layer uses the Transformer algorithm to calculate user requests and generate intermediate KV Cache data. Then, the KV Cache management layer improves cross-request reuse and cache management efficiency through a Radix tree. The specific access to the KV Cache relies on the collaborative storage mechanism between the GPU and CPU. The system pre-allocates a fixed-length buffer pool on the GPU side to store frequently accessed token cache data, while maintaining a long-term cache on the CPU side to save historical tokens. During this process, the GPU compression and decompression algorithms proposed in this application are executed by the GPU during heterogeneous KV Cache reading and loading, respectively. The method proposed in this invention can further eliminate GPU memory usage by fusing it with a general multiplication operator, thereby achieving better performance. It is worth noting that the technical solution of this application is decoupled from the model structure. Any model KV Cache based on the Transformer structure can use the technical solution of this application to perform lightweight KV Cache compression, thereby accelerating large model inference.
[0100] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps.
[0101] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data compression system for large language models disclosed in an embodiment of this application.
[0102] The processing unit 701 is used to perform exponential sparsification processing on the input key-value cache tensor to generate a sparse exponential high-order plane and a dense mantissa low-order plane; the sparse exponential high-order plane is used to represent the sparse tensor of the key-value cache tensor, and the dense mantissa low-order plane is used to represent the dense tensor of the key-value cache tensor.
[0103] The generation unit 702 is used to perform hybrid vectorization compression on sparse tensors to generate a compressed data matrix; the data matrix includes a column index matrix and a value matrix.
[0104] Storage unit 703 is used to store dense tensors, column index matrices and value matrices according to a preset tensor data structure, and to decompress them when needed to restore the data corresponding to the original key-value cache tensors.
[0105] For example, the compression system further includes: a decomposition unit 704;
[0106] Decomposition unit 704 is used to divide the key-value cache tensor into multiple compressed blocks according to the set dimensions of the large language model, and to perform exponent and mantissa decomposition on the floating-point elements in each compressed block to obtain the corresponding floating-point exponent part, floating-point mantissa part and floating-point sign part.
[0107] Decomposition unit 704 is also used to perform Zigzag-style remapping on the decomposed floating-point exponent part, and decompose the remapped exponent part into high-order sparse bits and low-order dense bits.
[0108] For example, the compression system further includes: an acquisition unit 705;
[0109] The generation unit 702 is specifically used to splice together the high-order sparse bits of different compressed blocks to generate a sparse exponential high-order plane.
[0110] The acquisition unit 705 is used to concatenate the low-order dense bits, floating-point sign part and floating-point mantissa part of the same compressed block to obtain the low-order dense plane of any compressed block; and to combine the low-order dense planes of different compressed blocks to obtain the dense mantissa low-order plane.
[0111] For example, the compression system further includes: a compression unit 706;
[0112] Unit 705 searches any compressed block, determines the range of floating-point elements within that compressed block, and obtains the starting and ending values of the range. The ending value is obtained from the starting value and a preset floating-point parameter. The range is... , The starting value of the interval. Preset floating-point parameters;
[0113] Compression unit 706 is used to symmetrically compress the floating-point exponent portion to the lower-order interval using a remapping function, so that the higher-order sparse bits present a continuous zero region; wherein, the remapping function is... , It is the literal value of the corresponding exponent part in the floating-point exponent part.
[0114] For example, the shape of the compressed block is ,in To set the number of dimensions, The number of compressed blocks within each defined dimension, with each compressed block containing 512 floating-point elements.
[0115] For example, the compression system includes:
[0116] The acquisition unit 705 is specifically used to scan according to the row arrangement of the sparse tensor, extract the non-zero values of each row and the column index of the corresponding non-zero values of each row; the column index is used to represent the position of the non-zero value of the sparse tensor in each row.
[0117] The acquisition unit 705 is also used to process the non-zero values or column indices of each row in parallel to obtain the column index matrix and the value matrix; wherein, the number of rows of each matrix corresponds to the number of rows of the sparse tensor, the number of columns of each matrix is related to the number of non-zero values in each row, and the value matrix is used to record the corresponding non-zero values in the sparse tensor.
[0118] For example, the compression system includes:
[0119] Storage unit 703 is specifically used to store dense tensors, column index matrices, and value matrices using a dual-path encoding structure; wherein the dual-path encoding structure includes a trunk path and an overflow path;
[0120] The main path uses a fixed-column-width three-dimensional tensor structure to store the column indices of non-zero values in the column index matrix and the corresponding column index values in the value matrix;
[0121] The overflow path is used to store non-zero values exceeding a fixed column width in the column index matrix and value matrix. The overflow path includes row-level metadata, overflow column index, and overflow value; the row-level metadata is used to record the number of overflows in each row.
[0122] The acquisition unit 705 is specifically used to fuse the data of the main path with the dense tensor to obtain the main path data; and to combine the data in the overflow path to obtain the overflow path data.
[0123] Storage unit 703 is specifically used to compress and store the main path data and the overflow path data.
[0124] For example, the compression system includes:
[0125] The processing unit 701 is specifically used to recover the sparse exponential high-order plane by decoding the trunk path data in parallel and combining it with the overflow path data.
[0126] Please refer to the following: Figure 8 The schematic diagram of a data compression device for large language models disclosed in this application includes:
[0127] Central processing unit 801, memory 805, input / output interface 804, wired or wireless network interface 803, and power supply 802;
[0128] Memory 805 is either a short-term storage memory or a persistent storage memory;
[0129] The central processing unit 801 is configured to communicate with the memory 805 and execute instructions stored in the memory 805 to perform the aforementioned operations. Figures 1 to 3 Data compression method for large language models in any of the embodiments shown.
[0130] This application also provides a chip system, which includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the aforementioned... Figures 1 to 3 Data compression method for large language models in any of the embodiments shown.
[0131] This application also provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned actions. Figures 1 to 3 Data compression method for large language models in any of the embodiments shown.
[0132] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figures 1 to 3 Data compression method for large language models in any of the embodiments shown.
[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0134] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.
[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A data compression method for large language models, characterized in that, The method includes: The input key-value cache tensor is subjected to exponential sparsification to generate a sparse exponential high-order plane and a dense mantissa low-order plane; the sparse exponential high-order plane is used to characterize the sparse tensor of the key-value cache tensor, and the dense mantissa low-order plane is used to characterize the dense tensor of the key-value cache tensor. The sparse tensor is subjected to hybrid vectorization compression to generate a compressed data matrix; the data matrix includes a column index matrix and a value matrix. The dense tensor, the column index matrix, and the value matrix are stored according to a preset tensor data structure, and decompressed when needed to restore the data corresponding to the original key-value cache tensor.
2. The data compression method for large language models according to claim 1, characterized in that, The exponential sparsity processing includes: The key-value cache tensor is divided into multiple compressed blocks according to the set dimensions of the large language model, and the floating-point elements in each compressed block are decomposed into exponent and mantissa to obtain the corresponding floating-point exponent part, floating-point mantissa part and floating-point sign part. The decomposed floating-point exponent is remapped using Zigzag style, and the remapped exponent is decomposed into high-order sparse bits and low-order dense bits.
3. The data compression method for large language models according to claim 2, characterized in that, The generation of the sparse exponent high-order plane and the dense mantissa low-order plane includes: The high-order sparse bits of different compressed blocks are spliced together to generate the sparse exponent high-order plane. The low-order dense bits, floating-point sign part, and floating-point mantissa part of the same compressed block are concatenated to obtain the low-order dense plane corresponding to any compressed block; the low-order dense planes of different compressed blocks are combined to obtain the dense mantissa low-order plane.
4. The data compression method for large language models according to claim 2, characterized in that, The Zigzag style remapping includes: Search any compressed block, determine the range of the floating-point element set within that compressed block, and obtain the starting value and ending value of the range; wherein, the ending value of the range is obtained from the starting value of the range and a preset floating-point parameter; the range is... The The starting value of the interval, the The preset floating-point parameters; The floating-point exponent portion is symmetrically compressed to the lower-order interval using a remapping function, so that the higher-order sparse bits present a continuous zero region; wherein, the remapping function is... The It is the literal value of the corresponding exponent part in the floating-point exponent part.
5. The data compression method for large language models according to claim 2, characterized in that, The shape of the compression block is , wherein The number of dimensions for the defined dimension, the The number of compressed blocks within each defined dimension, with each compressed block containing 512 floating-point elements.
6. The data compression method for large language models according to claim 1, characterized in that, The step of performing hybrid vectorization compression on the sparse tensor to generate a compressed data matrix includes: The sparse tensor is scanned according to its row arrangement, and the non-zero values of each row and the column index of the corresponding non-zero values of each row are extracted; the column index is used to characterize the position of the non-zero values of the sparse tensor in each row. The non-zero values of each row or the column index are processed in parallel to obtain the column index matrix and the value matrix; wherein, the number of rows in each matrix corresponds to the number of rows in the sparse tensor, the number of columns in each matrix is related to the number of non-zero values in each row, and the value matrix is used to record the corresponding non-zero values in the sparse tensor.
7. The data compression method for large language models according to claim 6, characterized in that, The step of storing the dense tensor, the column index matrix, and the value matrix according to a preset tensor data structure includes: A dual-path encoding structure is used to store the dense tensor, the column index matrix, and the value matrix; wherein, the dual-path encoding structure includes a trunk path and an overflow path; The main path uses a fixed-width three-dimensional tensor structure to store the column indices of non-zero values in the column index matrix and the column index values corresponding to the column indices in the value matrix; The overflow path is used to store non-zero values exceeding the fixed column width in the column index matrix and the value matrix. The overflow path includes row-level metadata, overflow column index, and overflow value. The row-level metadata is used to record the number of overflows in each row. The main path data is obtained by fusing the data in the main path with the dense tensor; the overflow path data is obtained by combining the data in the overflow path. The main path data and overflow path data are compressed and stored.
8. The data compression method for large language models according to claim 7, characterized in that, The decompression includes: The sparse exponential high-order plane is recovered by decoding the backbone path data in parallel and combining it with the overflow path data.
9. A data compression system for large language models, characterized in that, The system includes: The processing unit is used to perform exponential sparsification processing on the input key-value cache tensor to generate a sparse exponential high-order plane and a dense mantissa low-order plane; the sparse exponential high-order plane is used to characterize the sparse tensor of the key-value cache tensor, and the dense mantissa low-order plane is used to characterize the dense tensor of the key-value cache tensor. A generation unit is used to perform hybrid vectorization compression on the sparse tensor to generate a compressed data matrix; the data matrix includes a column index matrix and a value matrix. The storage unit is used to store the dense tensor, the column index matrix, and the value matrix according to a preset tensor data structure, and to decompress them when needed to restore the data corresponding to the original key-value cache tensor.
10. A data reasoning system for large language models, characterized in that, The system includes: The inference engine is used to execute the attention mechanism Transformer algorithm and generate key-value cache tensors; The key-value cache (KV) management layer is used to manage the storage and loading of the key-value cache tensors; The heterogeneous storage layer includes GPU memory and CPU memory; the GPU memory is used to allocate a fixed-length buffer pool to store the data corresponding to the key-value cache tensor; the CPU memory is used to maintain a long-term cache to save the data corresponding to historical key-value cache tensors. The inference system for large language models applies the compression method as described in any one of claims 1-8 to perform compression or decompression operations when the key-value cache tensor is stored or loaded.