A lossless compression method and system for large-scale KV precision

Abstract

This application provides a lossless compression method and system for large-scale KV-precision models, relating to the field of data processing. The method includes: acquiring key-value cache data generated during the inference process of a large language model; performing high-dimensional feature extraction and similarity analysis on the key-value cache data to generate corresponding hash signatures; physically reordering the key-value cache data based on the hash signatures, so that key-value cache data with similar features are arranged adjacently in physical storage space; transmitting the reordered key-value cache data to a smart network interface card (NIC) via a high-speed interconnect interface; and using the smart NIC to call its integrated hardware lossless compression module to perform streaming compression on the reordered key-value cache data. The technical solution of this application utilizes hardware offloading and pipeline technology, significantly reducing transmission bandwidth consumption and overall system latency, providing key technical support for efficient inference of large models in resource-constrained edge environments.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a lossless compression method and system for large-scale KV precision. Background Technology

[0002] In recent years, large language models (LLMs) have made significant progress in the field of natural language processing, and their powerful reasoning and generation capabilities have been widely applied in scenarios such as intelligent customer service, text generation, machine translation, and code completion. However, current large language models (LLMs) heavily rely on cloud computing, resulting in pain points such as long response times, high bandwidth costs, and privacy issues. Edge inference is a promising solution to address these challenges by deploying large language models (such as edge servers, edge gateways, and mobile devices) close to the network edge of the data source. However, deploying large models in edge environments faces significant challenges due to limitations in computing, bandwidth, and storage resources of edge devices.

[0003] In edge inference scenarios, tensor parallelism is often employed, distributing model parameters and computational tasks across multiple nodes for collaborative execution. While this parallel mechanism effectively distributes the computational load, it also introduces significant cross-device communication of intermediate results, primarily the frequent transfer of Key-Value (KV) caches. Current large language models typically employ a Transformer (neural network architecture) architecture, requiring iterative computation of the attention matrix for input tokens (the basic units of text processing and generation). To avoid redundant computation, the system usually caches historical token information (i.e., key-value pair cache, KV cache). When the model employs tensor parallelism or pipelined parallelism for inference, the KV cache needs to be frequently exchanged between different computation nodes, leading to significant communication bandwidth consumption and system latency, becoming a core bottleneck in the edge large model inference process.

[0004] To address the aforementioned issues, existing methods have attempted to reduce key-value (KV) size through model quantization, key-value truncation, or compressed storage. However, these methods often result in a loss of accuracy. For example, quantization introduces numerical errors, and KV truncation disrupts the attention-dependent structure, leading to a decrease in inference performance. Therefore, these methods are not widely applicable in edge inference tasks requiring high accuracy.

[0005] Against this backdrop, it has been noted that some edge servers are equipped with smart network interface cards (NICs, such as Data Processing Units) that possess programmable network acceleration and hardware compression capabilities, enabling high-speed data transfer and compression operations without consuming host computing resources. This invention proposes a lossless KV compression and transmission scheme based on smart NICs. Before KV cache transmission, lossless compression is performed, and the KV data compression process is offloaded from the host CPU / GPU to the smart NIC. This fully utilizes the parallel data channels and streaming compression hardware of the smart NIC, significantly reducing the latency of KV cache transmission between edge nodes and lowering the TTFT during LLM inference.

[0006] However, compressing key-value (KV) data using smart NICs still faces several challenges. Smart NICs employ the DEFLATE compression algorithm, which is hardware-bound and cannot be modified. Its compression principle is based on LZ77 dictionary matching and Huffman coding, making it highly sensitive to local similarities in the data. Large language models typically organize KV data directly according to generation order or batch order, resulting in similar segments being physically scattered. The compression algorithm struggles to capture repeating patterns within a limited window, preventing the hardware compression module from reaching its full potential and achieving extremely high compression ratios. Furthermore, when compressing large amounts of KV cache, smart NICs generate unacceptable compression latency, failing to meet user needs. Summary of the Invention

[0007] The purpose of this invention is to address the problem of low hardware compression efficiency caused by poor local similarity of data in existing KV cache compression methods based on smart network cards, and to provide a lossless compression method and system for large-model KV.

[0008] The above-mentioned objective of this application is achieved through the following technical solution:

[0009] S1: Obtain key-value cache data generated during the inference process of a large language model;

[0010] S2: Perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures;

[0011] S3: Based on hash signature, physical reordering of key-value cache data is performed so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space;

[0012] S4: Transmit the reordered key-value cache data to the smart network card via the high-speed interconnect interface;

[0013] S5: By calling its integrated hardware lossless compression module through the smart network card, the reordered key-value cache data is streamed and compressed.

[0014] Optionally, step S2 includes:

[0015] The step of performing high-dimensional feature extraction and similarity analysis on the key-value cache data to generate a corresponding hash signature specifically includes:

[0016] Flatten the key-value cache data into a two-dimensional matrix, where each row represents a key-value vector;

[0017] Locality-sensitive hashing algorithm is used to calculate a fixed-length hash signature for each key-value vector;

[0018] The locality-sensitive hashing algorithm is a similarity hashing algorithm, which performs a dot product operation between the key-value vector and a pre-generated random hyperplane matrix, and generates a hash signature based on the sign bit of the operation result.

[0019] The random hyperplane matrix is ​​generated using a fixed random seed during initialization;

[0020] The hash signature is 64 bits long.

[0021] Optionally, step S3 includes:

[0022] Physical reordering of key-value cache data based on hash signatures includes:

[0023] Sort the hash signatures of all key-value cached data in ascending or descending order of numerical value to generate a global similarity index;

[0024] Based on the global similarity index, the original key-value cache data is physically rearranged, and a corresponding sorting index file is generated so that downstream nodes can restore the original order of the key-value cache after receiving the data.

[0025] Optionally, step S4 includes:

[0026] The smart network card is a coprocessor with integrated hardware compression and acceleration functions.

[0027] Optionally, step S5 includes:

[0028] The smart network interface card (NIC) calls its integrated hardware lossless compression module to perform streaming compression, specifically including:

[0029] The key-value cache data to be compressed is divided into blocks of fixed size to obtain data blocks;

[0030] Create independent input and output buffers for each data block;

[0031] By submitting data in blocks asynchronously, the data is submitted sequentially to the hardware compression engine of the smart network card to obtain compressed blocks.

[0032] The hardware compression engine uses the standard DEFLATE lossless compression algorithm and combines it with the smart network card hardware acceleration module to achieve high-speed streaming.

[0033] The DEFLATE algorithm consists of two parts: the LZ77 sliding window matching module and the Huffman entropy coding module. The LZ77 sliding window matching module is used to scan the byte sequence of the data block, identify and remove historically occurring identical or similar data that exceeds a preset threshold. The Huffman entropy coding module is used to entropy code the symbols of the data block, assigning short codes to high-frequency symbols and long codes to low-frequency symbols.

[0034] Each compressed block has a header field compatible with the compression format and a tail check field based on a checksum algorithm automatically inserted at the beginning and end to determine whether the current block position is the first or last data block and trigger the corresponding head and tail data writing operation.

[0035] While the hardware compression engine processes the current data block, the next data block to be compressed is loaded in parallel.

[0036] Optionally, the hardware compression engine also includes a context self-healing mechanism. When the compression context is detected to be idle or an execution exception occurs, the context restart operation is automatically performed to reactivate the hardware compression engine and continue compression from the most recently successfully processed data blocks, thus achieving breakpoint resume compression.

[0037] A large-scale KV precision lossless compression system, the system comprising: a similarity hash sorting enhancement module on the server side and a DEFLATE lossless compression module on the smart network interface card side;

[0038] The similarity hash sorting enhancement module is connected to the DEFLATE lossless compression module via a high-speed interconnect interface;

[0039] The similarity hash sorting enhancement module is used to obtain key-value cache data generated during the inference process of large language models;

[0040] The similarity hash sorting enhancement module is also used to perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures;

[0041] The similarity hash sorting enhancement module is also used to physically reorder key-value cache data based on hash signatures, so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space.

[0042] The similarity hash sorting enhancement module is also used to transmit the reordered key-value cache data to the smart network card through a high-speed interconnect interface;

[0043] The DEFLATE lossless compression module is used to call the integrated hardware lossless compression module through the smart network card to perform streaming compression on the reordered key-value cache data.

[0044] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a large-model KV precision lossless compression method.

[0045] A computer-readable storage medium storing instructions that, when executed, perform a large-model KV precision lossless compression method.

[0046] The beneficial effects of the technical solution provided in this application are:

[0047] By analyzing the statistical correlation of key-value (KV) data between different layers or tokens of the Transformer, a similarity-based KV data reordering mechanism is designed. This mechanism spatially adjacentizes similar segments, improving the local compressibility of the data and fully leveraging the potential of the smart network interface card's compression channel to achieve high compression ratios and high bandwidth utilization in edge large-model inference scenarios. Simultaneously, to reduce the high latency associated with data reordering and compression, this invention proposes a streaming KV cache compression mechanism. This mechanism compresses and transmits KV cache data in a high-speed pipeline, significantly improving compression speed. Attached Figure Description

[0048] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0049] Figure 1 This is a system overall structure diagram in the embodiments of this application;

[0050] Figure 2 This is a heatmap of the average KV values ​​before sorting in the embodiments of this application;

[0051] Figure 3 This is a heatmap of the sorted KV average values ​​in the embodiments of this application;

[0052] Figure 4 This is a comparison chart of the compression and sorting effects of the smart network cards in the embodiments of this application;

[0053] Figure 5 This is a schematic diagram of the electronic device structure in the embodiments of this application;

[0054] Figure 6 This is a diagram showing the sorted and compressed result in an embodiment of this application. Detailed Implementation

[0055] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0056] The embodiments of this application provide a lossless compression method with large model KV precision.

[0057] Please refer to Figure 1 , Figure 1 This is a system overall structure diagram of a large-model KV precision lossless compression method according to an embodiment of this application, including:

[0058] S1: Obtain key-value cache data generated during the inference process of a large language model;

[0059] S2: Perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures;

[0060] S3: Based on hash signature, physical reordering of key-value cache data is performed so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space;

[0061] S4: Transmit the reordered key-value cache data to the smart network card via the high-speed interconnect interface;

[0062] S5: By calling its integrated hardware lossless compression module through the smart network card, the reordered key-value cache data is streamed and compressed.

[0063] Step S2 includes:

[0064] The step of performing high-dimensional feature extraction and similarity analysis on the key-value cache data to generate a corresponding hash signature specifically includes:

[0065] Flatten the key-value cache data into a two-dimensional matrix, where each row represents a key-value vector;

[0066] Locality-sensitive hashing algorithm is used to calculate a fixed-length hash signature for each key-value vector;

[0067] The locality-sensitive hashing algorithm is a similarity hashing algorithm, which performs a dot product operation between the key-value vector and a pre-generated random hyperplane matrix, and generates a hash signature based on the sign bit of the operation result.

[0068] The random hyperplane matrix is ​​generated using a fixed random seed during initialization;

[0069] The hash signature is 64 bits long.

[0070] As one embodiment, the SimHash algorithm is used to calculate a semantic signature for each key-value block. The SimHash algorithm maps a high-dimensional vector to a fixed-length hash signature through random hyperplane projection. This embodiment selects a 64-bit length to balance computational complexity and similarity resolution, and fixes the random hyperplane seed during the initialization phase to ensure that the signature results of different batches of data are reproducible.

[0071] Step S3 includes:

[0072] Physical reordering of key-value cache data based on hash signatures includes:

[0073] Sort the hash signatures of all key-value cached data in ascending or descending order of numerical value to generate a global similarity index;

[0074] Based on the global similarity index, the original key-value cache data is physically rearranged, and a corresponding sorting index file is generated so that downstream nodes can restore the original order of the key-value cache after receiving the data; the sorting index file includes: a sorting index file and a corresponding metadata file.

[0075] As one implementation, after the signature calculation is completed, the server generates a global similarity index by sorting the signature values ​​in ascending order, and then physically rearranges the KV data (i.e., the original key-value cache data) accordingly, outputting the sorted KV file and the corresponding index and metadata file. In this way, KVs with similar semantic features are physically adjacent, providing a more ideal data distribution for the subsequent compression stage and improving local similarity and compressibility.

[0076] Step S4 includes:

[0077] The smart network card is a coprocessor with integrated hardware compression and acceleration functions.

[0078] As one example, a smart network interface card (NIC) can be a device such as a DPU, KPU, or IPU.

[0079] Step S5 includes:

[0080] The smart network interface card (NIC) calls its integrated hardware lossless compression module to perform streaming compression, specifically including:

[0081] The key-value cache data to be compressed is divided into blocks of fixed size to obtain data blocks;

[0082] Create independent input and output buffers for each data block;

[0083] By submitting data in blocks asynchronously, the data is submitted sequentially to the hardware compression engine of the smart network card to obtain compressed blocks.

[0084] The hardware compression engine uses the standard DEFLATE lossless compression algorithm and combines it with the smart network card hardware acceleration module to achieve high-speed streaming.

[0085] The DEFLATE algorithm consists of two parts: the LZ77 sliding window matching module and the Huffman entropy coding module. The LZ77 sliding window matching module is used to scan the byte sequence of the data block, identify and remove historically occurring identical or similar data that exceeds a preset threshold. The Huffman entropy coding module is used to entropy code the symbols of the data block, assigning short codes to high-frequency symbols and long codes to low-frequency symbols.

[0086] As one implementation, to ensure the consistency and compatibility of the compressed stream, a Zlib-compatible header and an Adler32 checksum tail field are automatically inserted at the beginning and end of each compressed block. When the compression context detects the first or last block, the system automatically determines whether the current block position is the first or last data block, and writes the compressed stream start identifier or end checksum field accordingly, triggering the corresponding header and tail data write operations.

[0087] Each compressed block has a header field compatible with the compression format and a tail check field based on a checksum algorithm automatically inserted at the beginning and end to determine whether the current block position is the first or last data block and trigger the corresponding head and tail data writing operation.

[0088] As one implementation, to ensure the consistency and compatibility of the compressed stream, each compressed block automatically inserts a header field compatible with the compression format and a tail checksum field based on a checksum algorithm at the beginning and end. When the compression context detects the first or last block, the system automatically determines whether the current block position is the first or last data block, and writes the compressed stream start identifier or end checksum field accordingly, triggering the corresponding header and tail data write operations.

[0089] While the hardware compression engine processes the current data block, the next data block to be compressed is loaded in parallel.

[0090] As one embodiment, the smart network interface card (NIC) integrates a hardware-level DEFLATE lossless compression module. This module runs on the smart NIC hardware platform, utilizing the NVIDIA DOCA framework to access underlying compression hardware resources and achieve high-speed execution of standard DEFLATE compression. During system initialization, a compression context is created, a memory mapping creation operation is invoked to establish a shared physical memory mapping region between the host and the device, and a device registration operation is performed to add this physical memory region to the list of devices accessible to the device. Input and output buffer objects are established using the DOCA buffer interface, corresponding to uncompressed data and compressed results, respectively. The compression control module divides the input data into multiple blocks of a fixed size of 1.875MB, allocating an independent buffer to each block to achieve segmented streaming processing of long-sequence KV data. During the compression task submission phase, the system invokes an asynchronous compression task submission operation to initiate a task request to the hardware compression unit; during task execution, the main thread can continue loading the next batch of data, thus achieving parallel computation and I / O. After compression is complete, the callback function writes the result to the output file and automatically switches buffers to enter the next compression cycle.

[0091] As one embodiment, the hardware compression module of the smart network interface card (NIC) combines LZ77 dictionary matching with Huffman coding to achieve efficient encoding by detecting duplicate segments in the data. Because the server-side SimHash sorting concentrates similar data sets, the smart NIC can quickly match more repeating patterns within a local window, significantly reducing the compression ratio and coding redundancy. The compression result is output in a streaming manner, bypassing host memory and sent directly to downstream computing or storage nodes via the smart NIC, reducing transmission latency and improving bandwidth utilization. The system architecture supports multi-threaded parallel task scheduling and breakpoint resume, automatically restarting and resuming operation when a device error is detected, ensuring reliability under prolonged high loads.

[0092] The hardware compression engine also includes a context self-healing mechanism. When the compression context is detected to be idle or an execution exception occurs, the context restart operation is automatically performed to reactivate the hardware compression engine and continue compression from the most recently successfully processed data blocks, thus achieving breakpoint resume compression.

[0093] As one implementation example, the core of streaming DEFLATE lies in its asynchronous chunking and continuous compression mechanism. After the system chunks data, each chunk is submitted to the hardware compression engine via an asynchronous interface. Task execution and data loading are performed in parallel, and the output is automatically rearranged according to the sequence number to ensure the correct order of the compression results. The system has a context self-healing mechanism. When the compression context is detected to be idle or an execution exception occurs, the system automatically performs a context restart operation, reactivates the hardware compression engine, and continues compression from the most recently successfully processed data block, achieving breakpoint resume compression. The performance monitoring module records the compression ratio, throughput, and latency data in real time, dynamically adjusts the block size and concurrency, and ensures that the system can maintain stable and efficient operation under long-term high load conditions.

[0094] A large-scale KV precision lossless compression system, the system comprising: a similarity hash sorting enhancement module on the server side and a DEFLATE lossless compression module on the smart network interface card side;

[0095] The similarity hash sorting enhancement module is connected to the DEFLATE lossless compression module via a high-speed interconnect interface;

[0096] The similarity hash sorting enhancement module is used to obtain key-value cache data generated during the inference process of large language models;

[0097] The similarity hash sorting enhancement module is also used to perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures;

[0098] The similarity hash sorting enhancement module is also used to physically reorder key-value cache data based on hash signatures, so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space.

[0099] The similarity hash sorting enhancement module is also used to transmit the reordered key-value cache data to the smart network card through a high-speed interconnect interface;

[0100] The DEFLATE lossless compression module is used to call the integrated hardware lossless compression module through the smart network card to perform streaming compression on the reordered key-value cache data.

[0101] As one example, Figure 1 This paper demonstrates the overall structure of the edge large model tensor parallel KV precision lossless compression and transmission system based on a smart network interface card (NIC) proposed in this invention. The system mainly consists of two parts: a similarity hash sorting enhancement module on the server side and a DEFLATE lossless compression module on the smart NIC side. The two work together through a high-speed interconnect interface to achieve efficient lossless compression and transmission of KV cache during large model inference.

[0102] In one embodiment, in terms of system function division, the server side is mainly responsible for high-dimensional feature extraction and similarity reordering of KV data, while the smart network card side is responsible for performing streaming hardware compression and data transmission. The server-side module reads the KV cache and metadata of each layer and header from the inference engine, flattens and standardizes the high-dimensional tensor, so that the data is arranged continuously in the storage space.

[0103] As one example, the overall system workflow is as follows: The server receives and analyzes the KV cache (KV Cache or key-value cache) in real time, performs similarity hash signature calculation and similarity sorting, and transmits the sorted data to the smart network interface card (NIC) via a high-speed channel. The smart NIC performs hardware-level DEFLATE compression and streams the results. Downstream nodes decompress the data as needed and restore the original KV order using the sorting index, ensuring complete lossless data accuracy. Through this hierarchical collaborative mechanism, the system fully leverages the server's high-dimensional feature analysis and sorting optimization capabilities and the smart NIC's hardware-accelerated compression performance, achieving a significant improvement in KV compression ratio and transmission efficiency, while reducing communication bandwidth usage and system latency without altering the model structure.

[0104] Similarity Hash Sorting Enhancement Module: Without changing the main function of the smart network card, this invention designs a similarity hash sorting enhancement module. This module is mainly used to improve compression efficiency. It is located upstream of the device compression and is responsible for preprocessing and sorting KV data. The similarity hash algorithm can be considered as a similarity encoding method based on high-dimensional vector random hyperplane projection. Its core idea is to map arbitrary dimension vectors to fixed-length hash signatures, so that the distance between similar vectors in the hash space becomes smaller.

[0105] This module utilizes a similarity hashing algorithm to perform hash signatures and fast similarity estimation on high-dimensional vectors. This is used to discover potential local feature similarities (semantic relevance) in large-scale key-value (KV) datasets, thereby improving data compressibility at the physical level. The similarity hashing algorithm is an improved form of Locality Sensitive Hashing (LSH). It maps a high-dimensional real space to a fixed-length hash signature through random hyperplane projection, resulting in smaller Hamming distances between similar vectors and larger distances between dissimilar vectors in the hash space.

[0106] This invention utilizes this characteristic to perform signature calculation and similarity sorting on KV data. In the process, the KV tensor is first flattened into a two-dimensional matrix, with each row representing a KV vector to be compressed. SimHash calculates the sign direction of the vector on each hyperplane by multiplying it with a pre-generated random hyperplane matrix (1 for greater than zero and 0 for less than zero). The sign result is then packaged into a 64-bit integer. After the signature calculation of all vectors is completed, they are sorted by the quicksort algorithm to generate an index table, and the original KV data is rearranged according to the index.

[0107] from Figure 2 and Figure 3 A comparison of the heatmap results of the KV average values ​​before and after sorting clearly shows that the KV data before sorting exhibits a discrete distribution across different layers and token dimensions. The color blocks in the heatmap change frequently and are not continuous, indicating that the original KV vectors are semantically similar but scattered in physical space; while after SimHash similarity sorting ( Figure 3 The color distribution is significantly more concentrated and the local continuity is enhanced, indicating that similar KV data are effectively clustered into adjacent positions. This change reflects the core function of the SimHash module in this invention: by generating 64-bit semantic signatures through random hyperplane projection and rearranging them in ascending order according to the signature values, the high-dimensional KV vectors are reorganized according to similarity, which significantly improves the local correlation of the data. This provides more continuous repeating segments for LZ77 dictionary matching at the smart network card end, significantly enhancing the matching efficiency and compression ratio in the DEFLATE compression stage.

[0108] After sorting, similar data segments are physically closer together, which makes it easier for the DEFLATE compression algorithm to find repeating sequences in the LZ77 matching stage, thus improving the compression ratio. Test results show that SimHash sorting can reduce the overall compression ratio of the device's compression module by about 30% to 50%.

[0109] Furthermore, to ensure the stability of repeated experiments, this invention fixes the random seed during the SimHash initialization phase. This ensures that both the signature result and the sorting order are repeatable, avoiding fluctuations in the compression ratio of different batches of KV data. The calculation process of this module is entirely vectorized, and it only takes a few seconds to process millions of KV data points, demonstrating high scalability and versatility.

[0110] In one embodiment, to verify the performance of the proposed lossless KV compression system based on a smart network interface card (NIC), a system prototype was implemented in a real-world environment and its performance was evaluated. Taking a Data Processing Unit (DPU, a smart NIC device) as an example, the evaluation was conducted on a server equipped with a DPU. The method of this invention was run, using KV cache data generated by the GPT2-XL model as input, with a total data volume of 1.595 GB. All compression algorithms used were the standard DEFLATE, and all maintained completely lossless precision. Two comparison schemes were included: 1. A DPU compression scheme without similarity sorting; 2. A DPU optimized compression scheme using similarity hash sorting.

[0111] Evaluation results as follows Figure 4 As shown, the unsorted DPU compression scheme completes the compression of 1.595GB of KV data in 1.9 seconds, resulting in a compressed file of approximately 1.356GB, a compression ratio of approximately 85.01%, and a compression throughput of approximately 0.8GB / s. This scheme can significantly reduce CPU usage and achieve hardware-level streaming compression, but due to the local differences in KV data, the matching window utilization of the DEFLATE algorithm is low, and there is still room for further reduction in the compression ratio. After enabling the similarity hash sorting enhancement module of this invention, a 64-bit hash signature is first calculated for each KV vector and rearranged according to similarity, making similar KV data more physically adjacent, thereby improving the duplicate data matching rate of DEFLATE. The sorting stage takes approximately 0.1 seconds; subsequently, during DPU streaming compression, the compression time remains within 1.9 seconds, resulting in a compressed file of only approximately 497MB, with the compression ratio reduced to 31.15%. Combining sorting and compression, the total time is approximately 2 seconds; the overall efficiency is better than the traditional software scheme, while the compression ratio is significantly reduced (by more than 53%).

[0112] In one embodiment, the present invention was repeatedly tested under the same environment. The results showed that the compression ratio after sorting remained within the range of 35% ± 5%, while the unsorted scheme remained within 85% ± 5%. The technical means of the present invention has stable and repeatable improvement in compression ratio. Below are two examples from the experiment.

[0113] In one embodiment, the present invention employs the following hardware and software configuration and testing platform:

[0114] CPU: Intel® Xeon® Platinum8341C (48 cores, 3.00GHz).

[0115] Memory: 256GB DDR4.

[0116] Operating system: Ubuntu 22.04.5 LTS.

[0117] Server-side: Used to perform similarity hash sorting.

[0118] Smart network interface card device: DPU Mellanox BlueField-2.

[0119] DPU driver and compression engine: Uses the hardware DEFLATE compression unit that comes with the BlueField-2 firmware.

[0120] Model source: KV Cache generated during GPT2-XL inference process.

[0121] Test data size: 1.595GB KV cache.

[0122] Compression algorithm: Standard DEFLATE.

[0123] Precision maintenance: Yes.

[0124] The comparison of the two schemes is shown in Table 1 below:

[0125] Table 1

[0126]

[0127] Experimental procedure:

[0128] Step 1: Prepare KVCache data, i.e. cached data.

[0129] Run GPT2-XL to generate a full 48-layer KVCache on a fixed input text sequence, and merge them into a 1.595GB single file of KV data (keeping the KV layout consistent and ensuring that the input data of the two schemes are completely consistent).

[0130] Step 2: Execute Plan A (Unsorted): Divide the original KV data into blocks (1.875MB); use DPU to directly execute streaming DEFLATE; record compression time, result file size, and throughput.

[0131] Step 3: Execute Scheme B (Similarity Hash Sorting + Compression of this Invention): Flatten each KV vector into a one-dimensional vector; calculate a 64-bit similarity hash for each vector; sort by signature in ascending order (sorting time recorded); generate a new KV permutation file; divide the new KV data into blocks (1.875MB); transmit to the DPU to perform the same streaming DEFLATE; record compression time, output size, and throughput.

[0132] The experimental results are shown in Table 2 below:

[0133] Table 2

[0134]

[0135] like Figure 6 As shown in the results analysis, the compression ratio decreased by 53.86% after sorting, from 85.01% to 31.15%, indicating that sorting greatly enhanced the local similarity of the data.

[0136] Technical logic: DEFLATE's LZ77 relies on "locally repeating sequences"; the original key-value pairs are not sorted → repeating segments are scattered → matching is impossible. After similarity hash sorting → similar key-value pairs are clustered → the LZ77 distance value decreases and the matching success rate increases.

[0137] Compression speed remained unchanged, proving that sorting does not increase the DPU load. Compression time before sorting: 1.9 seconds. Compression time after sorting: still within 1.9 seconds.

[0138] Note: The sorting phase is completed on the server, without the need for DPU processing. The DPU maintains full-speed hardware compression and does not affect real-time performance.

[0139] This application also discloses an electronic device. (See reference...) Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0140] The communication bus 502 is used to enable communication between these components.

[0141] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0142] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0143] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned lossless compression method for large model KV precision.

[0144] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0145] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A lossless compression method for large-scale KV models, characterized in that, The method includes the following steps: S1: Obtain key-value cache data generated during the inference process of a large language model; S2: Perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures; Step S2 includes: The step of performing high-dimensional feature extraction and similarity analysis on the key-value cache data to generate a corresponding hash signature specifically includes: Flatten the key-value cache data into a two-dimensional matrix, where each row represents a key-value vector; Locality-sensitive hashing algorithm is used to calculate a fixed-length hash signature for each key-value vector; The locality-sensitive hashing algorithm is a similarity hashing algorithm, which performs a dot product operation between the key-value vector and a pre-generated random hyperplane matrix, and generates a hash signature based on the sign bit of the operation result. The random hyperplane matrix is ​​generated using a fixed random seed during initialization; The hash signature is 64 bits long. S3: Based on hash signature, physical reordering of key-value cache data is performed so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space; S4: Transmit the reordered key-value cache data to the smart network interface card via a high-speed interconnect interface; Step S4 includes: The smart network card is a coprocessor with integrated hardware compression and acceleration functions. S5: The smart network card calls its integrated hardware lossless compression module to perform streaming compression on the reordered key-value cache data; Step S5 includes: The smart network interface card (NIC) calls its integrated hardware lossless compression module to perform streaming compression, specifically including: The key-value cache data to be compressed is divided into blocks of fixed size to obtain data blocks; Create input and output buffers for each data block; By submitting data in blocks asynchronously, the data is submitted sequentially to the hardware compression engine of the smart network card to obtain compressed blocks. The hardware compression engine uses the standard DEFLATE lossless compression algorithm and combines it with the smart network card hardware acceleration module to achieve high-speed streaming. The DEFLATE algorithm consists of two parts: the LZ77 sliding window matching module and the Huffman entropy coding module. The LZ77 sliding window matching module is used to scan the byte sequence of the data block, identify and remove historically occurring identical or similar data that exceeds a preset threshold. The Huffman entropy coding module is used to entropy code the symbols of the data block, assigning short codes to high-frequency symbols and long codes to low-frequency symbols. Each compressed block has a header field compatible with the compression format and a tail check field based on a checksum algorithm automatically inserted at the beginning and end to determine whether the current block position is the first or last data block and trigger the corresponding head and tail data writing operation. While the hardware compression engine is processing the current data block, the next data block to be compressed is loaded in parallel. The hardware compression engine also includes a context self-healing mechanism. When the compression context is detected to be idle or an execution exception occurs, the context restart operation is automatically performed to reactivate the hardware compression engine and continue compression from the most recently successfully processed data blocks.

2. The lossless compression method for large-scale KV models as described in claim 1, characterized in that, Step S3 includes: Physical reordering of key-value cache data based on hash signatures includes: Sort the hash signatures of all key-value cached data in ascending or descending order of numerical value to generate a global similarity index; Based on the global similarity index, the original key-value cache data is physically rearranged, and a corresponding sorting index file is generated so that downstream nodes can restore the original order of the key-value cache after receiving the data.

3. A large-model KV precision lossless compression system, used to implement the large-model KV precision lossless compression method as described in any one of claims 1-2, characterized in that, The system includes: a similarity hash sorting enhancement module on the server side and a DEFLATE lossless compression module on the smart network card side; The similarity hash sorting enhancement module is connected to the DEFLATE lossless compression module via a high-speed interconnect interface; The similarity hash sorting enhancement module is used to obtain key-value cache data generated during the inference process of large language models; The similarity hash sorting enhancement module is also used to perform high-dimensional feature extraction and similarity analysis on key-value cache data to generate corresponding hash signatures; The similarity hash sorting enhancement module is also used to physically reorder key-value cache data based on hash signatures, so that key-value cache data with similar semantic features are arranged adjacent to each other in physical storage space. The similarity hash sorting enhancement module is also used to transmit the reordered key-value cache data to the smart network card through a high-speed interconnect interface; The DEFLATE lossless compression module is used to call the integrated hardware lossless compression module through the smart network card to perform streaming compression on the reordered key-value cache data.

4. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the large model KV precision lossless compression method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the lossless compression method for large model KV precision as described in any one of claims 1-2.

Images