Large language model reasoning acceleration method and system based on dynamic video memory compression and memory isomerism

Abstract

The invention discloses a big language model reasoning optimization method and system based on dynamic video memory compression and memory isomerism, and intelligent management of video memory resources is realized by integrating a dynamic compression strategy of KV Cache and a memory parallel architecture. The method comprises the following steps: 1) analyzing the spatial-temporal characteristics of the KV Cache in real time, adaptively selecting a quantization compression algorithm, a rarefaction algorithm or a low-rank decomposition algorithm, performing hierarchical storage based on attention head importance scores, keeping high precision of a core head, and implementing low-bit quantization on a secondary head; (2) the compressed inactive data are divided into a plurality of data blocks to be stored in a system memory, a parallel data channel group is established according to the number of physical channels, the compressed blocks are concurrently read through multiple channels during loading, and parallel decompression of a sparse matrix is accelerated through a GPU tensor core; and 3) constructing a KV Cache multiplexing mechanism and a parallel channel, and parallelizing a compression / decompression process and model calculation by adopting a hardware acceleration compression and asynchronous pipeline mechanism.

Description

Technical Field

[0001] The present invention belongs to the cross - technical field of deep learning and high - performance computing, and specifically relates to an inference acceleration optimization method for large language models (LLMs), especially aiming at the technical challenge of excessive video memory occupation of the key - value cache (KV Cache) during the inference process of language models.

[0002] Through the collaborative optimization of dynamic compression and memory heterogeneous storage, the present invention proposes a new video memory management scheme, covering multi - modal algorithms such as quantization compression, sparsification, and low - rank decomposition. Combining asynchronous pipelining operations with a hardware acceleration mechanism, it realizes the hierarchical and efficient utilization of video memory and system memory.

[0003] This method is applicable to natural language processing, multi - modal inference, and edge computing scenarios, and can effectively solve the video memory bottleneck problem of long - sequence inference, improving the model deployment ability in resource - constrained environments. Background Art

[0004] During the inference process of large language models, the video memory occupation of the key - value cache (KV Cache) that linearly increases with the sequence length has become the core problem restricting model efficiency. Taking mainstream models as an example, as the context window expands to thousands or even tens of thousands of tokens, the video memory requirements of the KV Cache far exceed the capacity of conventional GPU hardware, resulting in the forced interruption of inference tasks or a significant decline in performance. The existing technologies mainly have the following limitations: 1. Adaptability defects of static compression strategies. Traditional compression methods (such as fixed - bit quantization or undifferentiated sparsification) do not consider the spatio - temporal dynamic characteristics of the KV Cache. For example, the shallow attention heads of the model are sensitive to quantization errors, while the deep heads can tolerate a higher compression rate; at the same time, the recently generated KV Cache has a higher access frequency. Due to the lack of a dynamic hierarchical management mechanism in existing solutions, it is easy to cause accuracy loss or insufficient video memory utilization.

[0005] 2. Efficiency bottleneck of video memory - memory data exchange. Existing video memory optimization schemes (such as paging management or directly migrating uncompressed data) are limited by hardware bandwidth and transmission latency. The batch migration of uncompressed data will occupy a large amount of PCIe channel resources, resulting in the idle state of GPU computing units and significantly increasing the end - to - end inference latency. In addition, the frequent full - precision data back - transfer exacerbates the energy consumption burden of edge devices.

[0006] 3. Mismatch between sparse computing and hardware architecture. Although sparse technology can reduce video memory occupancy by pruning redundant elements, traditional sparse matrix formats (such as COO and CSR) are difficult to effectively map to the parallel computing mode of GPU tensor cores, resulting in a significant decrease in computing efficiency after decompression. This contradiction is particularly prominent in long sequence inference, offsetting part of the actual benefits of sparse compression with computational overhead.

[0007] 4. Lack of dynamic resource management in edge scenarios. In edge devices with limited video memory resources, existing solutions lack the ability of real-time resource monitoring and dynamic adjustment. Static memory swapping-out strategies cannot adapt to the dynamic changes in input sequence length and the number of concurrent tasks, easily causing frequent data migration or video memory overflow, leading to a decrease in service stability.

[0008] 5. Lack of collaborative optimization for multi-modal compression. Single compression techniques such as quantization, sparsification, and low-rank decomposition have their own advantages and disadvantages, but the current technical system lacks a multi-algorithm collaborative framework. For example, although quantization has low computational overhead, its compression ratio is limited, and low-rank decomposition is efficient for specific structured data but introduces additional computational load. Existing solutions fail to dynamically select the optimal compression combination according to the characteristics of the KV Cache, resulting in difficulty in breaking through the overall efficiency.

[0009] To address the above problems, there is an urgent need for a video memory management method that can dynamically perceive the characteristics of the KV Cache, support multi-algorithm collaborative optimization, and adapt to heterogeneous hardware architectures.

[0010] This patent aims to provide a new scalable technical path for high-concurrency, long-sequence, and low-resource scenarios by integrating dynamic feature analysis, mixed-precision quantization, sparse tensor core acceleration, memory virtual parallelism, and memory-video memory zero-copy transfer technologies. Summary of the Invention

[0011] The core of the present invention lies in constructing a large language model inference acceleration system with dynamic perception, hierarchical optimization, and hardware collaboration. Its overall design framework is as follows: The present invention proposes a large language model inference acceleration method and system based on dynamic video memory compression and memory heterogeneity. As shown in the appendix Figure 1 It is characterized by including: Main step 1) Determine the key-value cache (KV Cache) part that needs to be compressed during the inference process of the language model.

[0012] Main step 2) Compress the KV Cache according to a preset compression strategy. The compression strategy includes but is not limited to one or more of the following, as shown in the appendix Figure 2 And the appendix Figure 3 As shown: 1. Differentially quantize the KV Cache, implement a refined quantization strategy for different layers (e.g., use mild quantization for the first few layers to retain precision, and use aggressive quantization for the last few layers to achieve a higher compression ratio), and convert floating-point precision data into a low-bit representation (including but not limited to INT8 / 4-bit); 2. Sparsify the KV Cache by setting a threshold to remove elements with absolute values lower than the threshold, and store the non-zero elements and their coordinate indices in a sparse matrix format; 3. Perform low-rank decomposition on the KV Cache, decompose the original matrix into the product of two low-rank matrices, and retain the main feature dimensions.

[0013] For further optimization, the present invention utilizes the similarity of KV values of adjacent Tokens, and reduces redundancy by calculating the difference (Delta value) between adjacent Token KV Caches. Specifically, for the Delta values of Token i and i+1, their distribution is closer to zero, and compared with the original distribution, the Delta distribution is easier to compress.

[0014] For further optimization, the present invention also adopts arithmetic coding to optimize the numerical representation of the KV Cache, further reducing the storage and transmission overheads, including but not limited to: (a) Statistically analyze the numerical sequence of the KV Cache generated during the neural network inference process, and establish a numerical distribution probability model; (b) Generate a dynamic arithmetic coding table based on the probability model, and map the continuous numerical sequence to a variable-length binary code stream; (c) During the storage stage, use the arithmetic coding table to compress and encode the KV Cache numerical values to generate compressed data blocks; (d) During the loading stage, perform reverse arithmetic decoding on the binary code stream through a parallel decoder to restore the original numerical precision; (e) The dynamic update module periodically optimizes the probability model and the coding table according to the change of the historical numerical distribution.

[0015] The potential of arithmetic coding lies in that the traditional floating-point representation method has deficiencies in storage efficiency because each numerical value requires a large number of bytes. By adopting arithmetic coding, we can effectively reduce the storage requirements of the KV Cache. Arithmetic coding significantly reduces the number of bytes required to represent each numerical value by optimizing the binary representation of the numerical values. This method not only improves the compression ratio but also enables fast decoding during the inference process, thus maintaining high loading performance.

[0016] Main step 3) Store the compressed KV Cache data into the system memory (RAM) or a memory virtualization-based disk (RAM Disk). The following methods are adopted to accelerate the access speed of the KV cache in memory: 1. Divide the KV Cache data of the neural network into multiple data blocks and store them in the system memory in a compressed format; 2. Establish corresponding parallel data channel groups according to the number of physical channels of the memory; 3. When it is necessary to load the target KV Cache, simultaneously read multiple compressed data blocks from the memory through the parallel data channel groups; Perform parallel decompression processing on the read compressed data blocks (perform SIMD parallel decompression on multiple data blocks through the AVX-512 or AMX instruction set of the CPU; or start multiple decompression threads through the CUDA kernel function of the GPU, and each thread processes an independent data block), and splice the decompressed data into a complete KV Cache and transfer it to the video memory.

[0017] One of the design suggestions for the parallel implementation solution is as follows: Memory chunking strategy: A. Chunking dimension: (a) Time-step chunking: Split the KV Cache of the long sequence into N chunks according to time steps (e.g., each chunk has 128 tokens) and store them in different memory areas.

[0018] (b) Attention-head chunking: Allocate the KV Cache of different heads in multi-head attention to independent channels.

[0019] B. Metadata management: It is necessary to maintain a chunk index table (Block Mapping Table) to record the physical address and compression status of each chunk.

[0020] Parallel read acceleration: A. Multi-channel concurrency: (a) Use the SIMD instructions of the CPU (such as AVX-512, AMX) or the parallel threads of the GPU to initiate multiple memory read requests simultaneously.

[0021] (b) Example: If the memory controller supports 4 channels, 4 KV Cache chunks can be read simultaneously, and the theoretical bandwidth is increased by 4 times.

[0022] B. Prefetch mechanism: C Predict the next required chunk according to the sequence position of the generated token and decompress it in advance to the video memory buffer.

[0023] Compression and decompression optimization: A. Lightweight compression algorithm: Select low-latency algorithms such as Snappy or FPGA-accelerated Zstandard to ensure that the decompression time << memory read time.

[0024] B. Block-independent compression: Each memory block is compressed separately to avoid decompression serialization caused by cross-block dependencies.

[0025] In actual tests, we found that consumer-grade CPUs have limited memory channels (usually 4 channels). A server-grade CPU is required and it is best to form a dual-way configuration to give full play to its advantages. Therefore, in the specific implementation cases below, we used a dual-way Platinum CPU.

[0026] The KV Cache of the Transformer has temporal continuity and is suitable for block storage by time step or attention head, and is naturally suitable for parallel access.

[0027] The configuration method of the parallel data channel group may include: 1. Interleave the memory address space according to the number of channels; 2. Allocate an independent DMA controller for each physical memory channel; 3. Bind the correspondence between CPU cores and memory channels through the NUMA architecture.

[0028] Main step 4): When the model inference needs to call the KV Cache, read it from memory and decompress it to the video memory (VRAM). During the decompression process, asynchronous pipeline processing could have been adopted. When the GPU is executing the current computing task, the KV Cache required for the next time step is asynchronously transferred from memory to the video memory and decompressed through an independent thread.

[0029] Based on the above solution, through designs such as memory channel parallelization + compression algorithm differentiation + prefetching intelligence, a breakthrough improvement in the KV Cache loading speed can be achieved in a large memory environment. In actual tests, the inference throughput of the LLM has increased significantly, even up to 2 - 3 times or more, providing a new technical paradigm for the deployment of large-scale language models.

[0030] In addition, based on the present invention, a KV Cache reuse inference system can be further developed, such as: (a) Perform semantic analysis on the context of large language model inference requests to identify input features with the potential for repeated use and generate unique context identifiers; (b) Store the pre-computed KV Cache in compressed format in memory or RAMDISK; (c) Through a hardware-accelerated compression / decompression pipeline, load the selected KV Cache from memory to the GPU video memory and perform the following operations in parallel: - Asynchronously decompress and cache data using tensor cores - Directly map active context segments to the attention calculation unit - Reserve 5 - 10% of the video memory space for dynamically updating the high-frequency reuse cache.

[0031] Based on the present invention, a multi-model shared KV Cache knowledge management system based on a large-capacity memory can be further fabricated, which is characterized by including: (a) Deploy a distributed knowledge transfer network (KDN) on a server node equipped with ≥256GB DDR memory, and establish a shared KV cache pool; This KDN includes an intelligent prefetching subsystem, and its operating mechanism includes: 1. Monitor the changing trend of the attention pattern during model inference 2. When detecting a query topic shift, asynchronously load the KV cache blocks in the associated field 3. Achieve zero-copy transfer from memory to video memory through PCIe P2P DMA.

[0032] The implementation of multi-model compatibility in this shared KV cache pool includes: - Convert the positional encoding of different models into a unified relative position encoding format - Re-align the dimension differences of the hidden states of each model through tensor slicing and recombination - Attach model architecture metadata (number of layers / number of heads / dimension ratio) to each cache block.

[0033] (b) Map the input context to a 128-bit hash identifier through a semantic fingerprint engine, and generate a unified cache index key across models.

[0034] (c) The dynamic cache allocation module optimizes the storage strategy in real time according to the following parameters: - Similarity matrix of the attention head structures of each model - Exponentially decaying statistics of cache access frequencies - Bandwidth utilization between GPU video memory and DDR memory.

[0035] (d) Adopt a hierarchical compression strategy to differentially process KV Caches with different sharing levels: ① High-frequency shared blocks retain FP16 precision and are directly memory-resident ② Medium-frequency shared blocks apply 4-bit quantization + sparse coding compression ③ Low-frequency shared blocks are transferred to NVMe SSD and an LRU chained index is established.

[0036] In specific implementations, the following processes are also adopted: 1. Compression stage: Convert the KV Cache into a binary representation through encoding and store it in the system memory (RAM) or a disk based on memory virtualization (RAM Disk). 2. Decompression stage: Retrieve the binary representation according to the request, decode it, and load the decompressed KV Cache into the VRAM for inference.

[0037] In terms of specific details, its technical solution also includes the following key modules and methods: 1. Dynamic video memory compression engine (1) Spatiotemporal feature analysis module Time dimension analysis: Real-time track the access frequency and timestamp of the KV Cache, and divide the active and inactive regions. Identify the low-frequency access parts in the historical cache through the sliding window mechanism and mark them as candidates for eviction.

[0038] Space dimension analysis: Based on the importance score of attention heads (calculated through gradient backpropagation or output sensitivity), distinguish the core attention heads from the secondary heads. The core heads retain the original precision (FP16 / FP32), and the secondary heads enable quantization / sparsification as needed.

[0039] Dynamic decision-making algorithm: Construct a lightweight classification model (such as a decision tree or a small neural network), and based on context parameters such as the current video memory pressure, hardware computing power, and sequence length, real-time select the optimal compression strategy combination (quantization + sparsification or low-rank decomposition + lossless compression).

[0040] (2) Multimodal compression algorithm library Mixed-precision quantization: Implement dynamic bit-width quantization for the secondary head KV Cache, and dynamically calibrate the quantization interval based on the error feedback mechanism. Support modes such as INT8 / 4-bit, and retain the high-precision data of the core heads.

[0041] Sparsification compression: Adopt an element clipping strategy with adaptive threshold, combined with the block sparse format optimized by the GPU tensor core (such as the 2:4 sparse mode), and compress the non-zero element indices into bit masks.

[0042] Low-rank approximation: Implement incremental singular value decomposition (SVD) for the KV Cache layer with weak long-distance dependencies, dynamically retain the main component feature dimensions, and reduce the storage requirements.

[0043] Delta encoding and arithmetic compression: Utilize the continuity of the KV values of adjacent Tokens, calculate the difference (Delta), and further compress the redundant information through arithmetic encoding.

[0044] 2. Hierarchical storage management system (1)Video Memory - Memory Co - architecture Hierarchical storage strategy: Video memory layer: Retain active KV Cache and core header data to ensure zero - latency access to frequently - accessed data.

[0045] Memory layer: Store compressed inactive KV Cache, managed by memory - mapped files, supporting fast random access.

[0046] Disk layer (e.g., RAMDISK, optional): For ultra - long context scenarios, compress and persistently store extremely - low - frequency - accessed data, and load it on - demand through a pre - fetch mechanism.

[0047] (2)Dynamic eviction and pre - fetch mechanism Eviction strategy: When the video memory occupancy exceeds a preset threshold, compress and migrate inactive KV Cache to memory according to the LRU (Least Recently Used) strategy. Refer to the appendix Figure 6 as shown.

[0048] Pre - fetch strategy: Combining model decoding step prediction, when the GPU calculates the current token, asynchronously pre - fetch the compressed data that may be accessed in the next time step and decompress it into the video memory buffer.

[0049] 3. Computation - transmission parallelization pipeline (1)Asynchronous compression / decompression pipeline Independent execution threads: When the GPU computing unit processes the forward propagation of the current layer, perform the following operations in parallel through a dedicated computing stream (CUDA stream or CPU processor AMX instruction calculation): Compress the completed KV Cache and write it to memory; Pre - read the KV Cache data required for the next layer from memory and decompress it into the video memory buffer. The process in a specific case is as shown in the appendix Figure 4 as shown.

[0050] Hardware acceleration interface: Call GPU tensor core instructions (such as NVIDIA DPX instructions) or CPU processor AMX instructions to accelerate sparse matrix calculations, and use a dedicated compression hardware module (such as NVENC) to accelerate Zstandard / LZ4 algorithms.

[0051] (2)Mixed - precision computing scheduling Precision - aware computational graph: During model inference, dynamically identify KV Cache data streams of different precisions (FP16 / INT8 / sparse blocks), and automatically match the corresponding computational kernel functions to avoid precision conversion overhead.

[0052] Error Compensation Mechanism: For quantized and compressed data, a residual correction term is introduced during attention calculation, and the numerical deviation introduced by compression is compensated through linear interpolation.

[0053] 4. Edge Scene Adaptive Optimization (1)Resource Monitoring and Dynamic Adjustment Real-time Resource Dashboard: Continuously collect metrics such as video memory / memory usage rate, PCIe bandwidth occupancy rate, compression and decompression latency, etc.

[0054] Dynamic Parameter Adjustment: Automatically adjust the compression ratio (such as switching from INT8 to 4-bit), swap-out frequency, and prefetch window size according to the resource status to achieve a balance between latency and video memory occupancy. Refer to the appendix Figure 7 。

[0055] (2)Lightweight Compression Strategy Edge Optimization Algorithm Library: Provide a simplified version of the compression algorithm (such as only supporting quantization + Delta coding) for the characteristics of mobile GPUs to reduce computational overhead.

[0056] Energy Consumption Awareness Mode: In battery-powered scenarios, prefer low-power compression strategies (such as lossless compression instead of low-rank decomposition) to extend the device's battery life.

[0057] 5. System Integration Framework Plugin-based Architecture: Provide standard API interfaces to support seamless integration with mainstream inference frameworks (such as TensorRT-LLM), and extend its PagedAttention mechanism to support memory swap-out. Refer to the appendix Figure 8 。Plugin and Execution: 1. Adopt the block pre-filling technology to split the long prompt into multiple smaller blocks and allow parallel processing of the GPU and CPU to reduce inference latency; 2. Introduce hash-based prefix cache management to share the KV cache in multi-turn conversations.

[0058] Cross-platform Deployment: Adapt to different hardware through the abstraction layer and automatically enable corresponding acceleration instructions.

[0059] Technical Solution Process: 1. Input Sequence Processing → 2. KV Cache Generation → 3. Dynamic Feature Analysis → 4. Compression Strategy Decision → 5. Hierarchical Storage Execution → 6. Asynchronous Prefetching and Decompression → 7. Mixed Precision Calculation → 8. Output Generation → 9. Resource Monitoring and Feedback Regulation Through the collaborative effect of the above technical modules, this solution realizes the multi-objective optimization of video memory occupancy, computational efficiency, and model accuracy, providing a systematic solution for the efficient deployment of large language models. Beneficial Effects

[0060] Through the collaborative optimization mechanism of dynamic video memory compression and hierarchical storage, the present invention provides an efficient and flexible video memory management solution for large language model inference. Its core advantages are reflected in the following aspects: Efficient utilization of video memory resources: By dynamically perceiving the spatio-temporal characteristics of the KV Cache, combined with the hierarchical adaptation of multimodal compression algorithms (quantization, sparsification, low-rank decomposition), refer to the appendix Figure 5 , significantly reducing the video memory occupancy of a single task and supporting longer context sequence inference capabilities. The hierarchical storage architecture migrates inactive data to system memory or persistent storage, expanding the effective capacity of the video memory and avoiding task interruption or degradation caused by insufficient video memory in traditional solutions.

[0061] Balance between inference efficiency and computational overhead: The asynchronous pipeline design parallelizes compression / decompression operations and model calculations, effectively hiding compression and transmission delays through hardware acceleration interfaces (such as GPU tensor core instructions and CPU's AMX instructions) to ensure end-to-end inference efficiency.

[0062] The deep combination of sparsification compression and dedicated computing cores reduces the performance loss of sparse matrix operations and maintains a computational throughput close to that of the original dense matrix.

[0063] Guarantee of model accuracy controllability: Based on the mixed precision strategy of attention head importance scoring, it retains the high-precision expression of key features and suppresses the transmission and accumulation of quantization errors in the attention mechanism. The dynamic calibration mechanism and residual compensation technology adaptively adjust compression parameters to ensure that the output quality after compression is close to the inference result of the original model.

[0064] Wide adaptability to edge and heterogeneous hardware: The dynamic resource monitoring module real-time feedbacks the video memory, bandwidth, and computing power status, automatically switches compression strategies and storage levels, and adapts to diverse deployment scenarios from cloud GPUs to edge devices. Zero-copy transmission and memory mapping file technology reduce data migration overhead, support stable inference services in low-bandwidth environments, and improve the deployment feasibility of edge devices.

[0065] System scalability and compatibility: Compatible with mainstream inference frameworks (such as TensorRT-LLM) through a plugin architecture, it can be seamlessly integrated into existing model service pipelines, reducing the technical migration cost. Adapting to multi-platform acceleration instruction sets across hardware abstraction layers ensures the efficient execution of algorithms on heterogeneous devices such as NVIDIA and AMD.

[0066] Optimization of energy consumption and cost: By reducing video memory occupancy, it reduces the dependence on high-specification GPU hardware, supports running ultra-large-scale models on consumer-grade graphics cards, and reduces hardware procurement and operation and maintenance costs.

[0067] In summary, the technical solution of the present invention systematically solves the video memory bottleneck problem in large language model inference through the deep integration of dynamic compression decision-making, hierarchical storage architecture, and hardware co-acceleration. Its core value lies in achieving a multi-dimensional optimal balance among video memory occupancy, computing efficiency, and model accuracy, while supporting flexible deployment from the cloud to the edge, providing key technical support for the popularization application of large-scale language models. Description of the Drawings

[0068] Figure 1 : Overall system architecture diagram. This figure shows the overall architecture of the large language model inference acceleration system based on dynamic video memory compression and hierarchical storage. It includes the interaction relationships among the dynamic feature analysis module, multi-modal compression algorithm library, hierarchical storage manager, asynchronous pipeline scheduler, and resource monitoring module, as well as the data flow paths among video memory, memory, and disk.

[0069] Figure 2 : Dynamic compression decision flow chart. This figure describes the dynamic selection process of the compression strategy for the KV Cache. Starting from spatio-temporal feature analysis (importance scoring of attention heads, access frequency monitoring), combined with the real-time pressure of the video memory and the state of the hardware computing power, compression instructions for quantization, sparsification, or low-rank decomposition are generated through the decision model.

[0070] Figure 3 : Schematic diagram of the hierarchical storage structure. This figure shows the three-level storage system of video memory - memory - disk. The video memory layer stores active KV Cache and core attention head data, the memory layer stores compressed inactive data, and the disk layer (optional) manages the persistent cache for ultra-long contexts. The arrows indicate the data migration directions and triggering conditions.

[0071] Figure 4 : Asynchronous pipeline timing diagram. This figure shows the parallelization process of GPU computing and compression / transmission operations through the time axis. It includes the timing overlap relationships among the current computing task, prefetching of the next batch of data, compression and writing back to memory, etc., reflecting the pipeline optimization of CPU-GPU cooperation.

[0072] Figure 5 : Schematic diagram of mixed-precision KV Cache management. This figure compares the differential processing of core heads and secondary heads. The core heads retain high-precision data and reside in the video memory, while the secondary heads are migrated to the memory after quantization or sparsification compression. The dotted boxes indicate the data form changes before and after compression.

[0073] Figure 6 : Schematic diagram of sparse matrix compression and calculation. This figure shows the block sparse compression process optimized based on GPU tensor cores. It includes threshold clipping of the original KV Cache, encoding of the block sparse format (such as the 2:4 sparse mode), bit mask generation, and the execution mapping relationship of sparse matrix multiplication on tensor cores.

[0074] Figure 7 : Flowchart of Edge Device Adaptive Adjustment. This figure describes the dynamic resource management process in the edge scenario. Starting from the monitoring of video memory / memory occupancy rate, it triggers the closed-loop control logic of compression ratio adjustment, page-out frequency optimization, and power consumption mode switching, reflecting the adaptive decision-making mechanism.

[0075] Figure 8 : Schematic Diagram of System Integration Framework. This figure shows the integration method of the plug-in architecture and the mainstream inference framework. It includes the extended connection of the standard API interface, the compression algorithm library, and the original paging management module, as well as the design of the abstract adaptation layer across hardware platforms (NVIDIA / AMD / edge chips).

[0076] Specific Implementation Cases One of the specific implementation cases is that in the internal test of the intelligent triage system in a well-known hospital in eastern Zhejiang in April 2025, a medical large model based on a large model architecture with 70B parameters was deployed to analyze the medical record text and imaging reports of emergency patients in real time. The system uses a heterogeneous computing cluster, including inference nodes equipped with large-capacity memory and dedicated compression acceleration hardware, forming a core architecture for dynamic video memory management. The cluster consists of multiple devices, and the single-node implementation architecture of the specific devices is as follows: Hardware Environment: 1. GPU Nodes: NVIDIA RTX 4090 × 2 (Total video memory: 96GB, this 4090 is a version with modified video memory particles) 2. Memory System: 1TB DDR4 ECC Memory (16 configurations of 64GB) 3. Processor: Dual-way Intel Platinum Processor, with a total of 112 logical cores 4. Storage Acceleration Card: YFTK Acceleration Card (Integrated with Zstandard Hardware Coding and Decoding Engine) (Before using the technical solution of the present invention, the memory utilization rate was very low, and the CPU load was also very low. After using this solution, both have been significantly improved. The inference effect has also been significantly enhanced.) The Core Implementation Process is as follows: 1.1. Compression and Persistent Storage of KV Cache Delta Encoding Compression Based on Continuity Adjacent Token Similarity Analysis: Extract the horizontal continuity features of the KV Cache in the Token dimension, calculate the difference (Delta value) for the Key / Value matrix of adjacent Tokens, and generate a sparse Delta matrix.

[0077] Block-level Delta Encoding: Divide the Delta matrix into data blocks of a fixed size (e.g., 128×128), perform dynamic range statistics on the differences within each block, and use Huffman coding or arithmetic coding to further compress redundant information.

[0078] Binary Persistent Storage: Package the compressed Delta-encoded data with metadata (block size, encoding parameters) into a binary format and store it on a local disk or a remote storage system.

[0079] Hierarchical Differential Quantization Strategy Shallow Quantization Protection Mechanism: For the KV Cache in the first 1 / 3 layers of the model, adopt a conservative FP8 quantization strategy to retain more high-frequency detail information to maintain the accuracy of attention calculations.

[0080] Deep-layer Efficient Compression: Implement FP6 or INT8 quantization on the KV Cache of deep layers (such as the last 2 / 3 layers), and adjust the quantization interval in combination with dynamic calibration technology to achieve a higher compression ratio.

[0081] Mixed-precision Metadata Embedding: Mark the quantization modes and parameters of each layer at the head of the compressed data to ensure the accurate restoration of the original numerical distribution during decompression.

[0082] Arithmetic Coding Optimization Numerical Distribution Modeling: Based on the statistical characteristics of the KV Cache (such as Gaussian distribution or Laplace distribution), construct an adaptive probability model to optimize the symbol mapping table of arithmetic coding.

[0083] Hardware-accelerated Decoding: Utilize the parallel computing power of the GPU to split the arithmetic coding stream into multiple sub-blocks and improve throughput through multi-threaded concurrent decoding.

[0084] 1.2. Fast Loading and Decompression of Compressed Data Preloading and Decompression Pipeline On-demand Prefetching Mechanism: According to the prediction of the context length of the inference request, asynchronously prefetch the possibly required compressed KV Cache data from the disk / remote storage to the memory buffer.

[0085] Heterogeneous Accelerated Decompression: Call the CUDA kernel functions of the graphics card and the processor to jointly implement the parallelization of Delta decoding, dequantization, and arithmetic decoding.

[0086] KV Cache Sharding Strategy -- RAID 0-style Striped Storage: Split the 24GB KV Cache into 128KB blocks and cyclically distribute them to 8 memory channels.

[0087] 1.3. Dynamic Fusion and Selective Prefilling of Multiple KV Caches Cross-document Cache Hybrid Mechanism Prefix cache marking: Add a unique prefix identifier to the KV Cache generated for different documents to preserve the attention calculation boundaries of each document during cache mixing.

[0088] Cross-attention restoration: When merging multiple KV Caches, partially pre-fill the Token embedding layer in the connection area, recalculate the cross-document attention weights, and suppress semantic confusion.

[0089] Selective pre-fill optimization Token-level change detection: Dynamically identify sensitive areas that need to be recalculated by comparing the gradient change amount or attention score difference of adjacent Tokens.

[0090] Hierarchical partial pre-fill: Base layer (such as Embedding layer): Full pre-fill to ensure semantic coherence; Middle layer: Only locally pre-fill Tokens whose change rate exceeds the threshold; High layer: Reuse the historical KV Cache and compensate for the accuracy loss through residual connections.

[0091] Hybrid engine scheduling Standard mode: Directly reuse the complete KV Cache, suitable for scenarios that are sensitive to latency but have loose accuracy requirements; Enhanced mode: Activate selective pre-fill and cross-attention restoration for multi-document inference tasks that require high-precision output.

[0092] 1.4. Edge scenario adaptation optimization Bandwidth-aware compression strategy: In an old memory environment (such as DDR3 ECC memory), prefer high-compression algorithms (arithmetic coding + FP6 quantization) to reduce data transmission volume.

[0093] Dynamic cache eviction strategy: Dynamically adjust the retention duration of the KV Cache according to the video memory usage rate, perform LRU eviction on low-frequency accessed data, trigger secondary compression, and then swap it out to memory.

[0094] 1.5. Implementation process Data preparation stage: Perform Delta coding, hierarchical quantization, and arithmetic compression on the input documents respectively, and generate binary KV Cache for storage on disk.

[0095] Request processing stage: Load relevant compressed data from disk according to the query requirements, decompress it to the video memory through the GPU pipeline; Execute cross-document cache mixing and selective pre-fill, and restore cross-attention calculation; Invoke the mixed-precision computing kernel to complete the inference and output the results.

[0096] Resource recycling stage: Dynamically eliminate inactive KV Cache according to the video memory pressure, compress it and then migrate it to memory or the RAMDISK disk.

[0097] Through the synergistic effect of the above technical means, this embodiment significantly reduces the storage and transmission overhead of KV Cache, improves the utilization rate of DDR4 ECC memory, improves the utilization rate of the CPU processor, and at the same time ensures the inference accuracy and efficiency in the multi-document scenario, providing a feasible engineering path for the large-scale application of large language models.

[0098] Specific implementation case two This case is a dynamic KV Cache optimization practice for a large model system in the financial public opinion analysis scenario. During the internal test phase of a real-time public opinion monitoring platform of a financial institution, a large model based on a certain 671B architecture was deployed to analyze a large amount of news, financial reports and social media data. The system is equipped with a heterogeneous storage environment, including a GPU server cluster and a distributed memory pool: 1. Hardware configuration: (1) Computing nodes: An array of 3090 graphics cards with 24G video memory, supporting mixed-precision computing; (2) Virtual storage: RAM Disk based on memory virtualization; 2. Software modules: (1) Real-time monitoring agent: Continuously track the numerical distribution and access patterns of KV Cache; (2) Intelligent compression engine: An adaptive decision-making module integrating multi-strategy compression algorithms.

[0099] 3. The core implementation process is as follows: Scenario: Analysis of sudden negative news of a listed company.

[0100] (I). Dynamic analysis stage When the public opinion warning of "a certain technology giant is suspected of financial fraud" is input: A. Feature recognition: The numerical tracker finds that the variance of the Key vectors of the 5th - 8th layers of Transformer suddenly increases; B. Hotspot marking: Mark the attention heads related to fields such as "financial statements" and "audit opinions" as high-sensitivity areas; C. Strategy decision-making: Adopt 8-bit quantization for the first 3 layers, implement threshold sparsification for the middle layers, and perform low-rank decomposition for the last layer.

[0101] (II) Differential compression execution (1) Quantization of the first 3 layers: Retain the key numerical precision.

[0102] The Key vectors of core concepts such as "revenue recognition" and "cash flow" in A are retained with an error range of ±1%. B uses an asymmetric quantization algorithm to dynamically adjust the zero-point offset.

[0103] (2) Intermediate layer sparsification: Filter out noise information.

[0104] A sets an adaptive threshold: Elements with |Value| < (moving window mean × 0.3) are set to zero; B is stored in Block-CSR format, and 1 coordinate index is recorded for every 32 elements.

[0105] (3) Final layer low-rank decomposition: Extract core features.

[0106] A decomposes the Value matrix related to market value prediction into a product form of rank 64; B retains the top 10% of the singular values to construct an approximate matrix.

[0107] (III) Heterogeneous storage management A High-frequency data: The analysis results of the current quarter's financial report are stored in the hot zone; B Medium-frequency data: Industry comparison data is compressed and stored in the standard area of the memory pool; C Low-frequency data: The historical case library is stored in the RAM Disk virtual disk in a decomposed form (constructed through the imdisk driver).

[0108] (IV) On-demand decompression and reconstruction When the model needs to trace back the history of "audit agency change": (1) Concurrently load 3 types of compressed blocks from the memory pool: A. Block 1: Quantified audit key points index; B. Block 2: Features of violation cases stored in sparse form; C. Block 3: Low-rank decomposed industry risk matrix.

[0109] (2) Synchronously execute on the GPU: A. The accuracy of the quantified data is restored through a lookup table; B. Skip zero-value calculations during sparse block reconstruction; C. Expand the low-rank matrix through outer product.

[0110] 4. Implementation details of key technologies (1) Dynamic decision-making mechanism - Construct a three-layer decision tree: A. Feature dimension > 1024 and entropy value > 5.0 → Enable low-rank decomposition; B. Proportion of zero values > 65% → Trigger structured sparsity; C. Standard deviation of numerical distribution < 0.1 → Implement non-uniform quantization.

[0111] (2)Memory storage optimization - Adopt an interleaved storage strategy: A. Scatter the quantized data in 4 memory channels in 64KB blocks; B. Establish a B+ tree for fast positioning of sparse indexes; C. Store the U / V factors of the low-rank matrix separately to improve the read parallelism.

[0112] (3)Decompression acceleration design - AMX heterogeneous computing: A. Skip zero-value calculation units during sparse matrix reconstruction; B. Fuse the execution of quantized data reconstruction and matrix multiplication; C. Use a block pipeline for low-rank outer product calculation.

[0113] 5. Implementation effect verification (1)Performance improvement. When parsing a hundred-page PDF financial report, the video memory occupancy is significantly reduced; (2)Resource optimization. Improve the memory storage efficiency through hierarchical compression.

[0114] In summary, the present invention has been effectively verified in multiple specific implementation cases.

Claims

1. A large language model inference acceleration method and system based on dynamic video memory compression and memory heterogeneity, characterized by including: Determine the key-value cache (KV Cache) part that needs to be compressed during the inference process of the language model; Compress the KV Cache according to a preset compression strategy; Store the compressed KV Cache data in the system memory (RAM) or a disk based on memory virtualization (RAMDisk); When the model inference needs to call the KV Cache, read it from the memory and decompress it to the video memory (VRAM); The compression strategy includes one or more of the following:

1. Differentially quantize the KV Cache and implement a refined quantization strategy for different layers (e.g., use mild quantization for the first few layers to retain precision, and use aggressive quantization for the last few layers to achieve a higher compression ratio); 2. Sparsify the KV Cache by setting a threshold to remove elements with absolute values lower than the threshold, and store the non-zero elements and their coordinate indices in a sparse matrix format; 3. Perform low-rank decomposition on the KV Cache, decomposing the original matrix into the product of two low-rank matrices to retain the main feature dimensions.

2. The system according to any one of the preceding claims, characterized in that, The following process is adopted:

1. Compression stage: Convert the KV Cache into a binary representation through encoding and store it in the system memory (RAM) or a disk based on memory virtualization (RAM Disk); 2. Decompression stage: Retrieve the binary representation according to the request, decode it, and load the decompressed KV Cache into the VRAM for inference.

3. The system according to any one of the preceding claims, characterized in that, Utilize the similarity of KV values of adjacent Tokens to reduce redundancy by calculating the difference (Delta value) between adjacent Token KV Caches.

4. The system according to any one of the preceding claims, wherein Adopt arithmetic coding to optimize the numerical representation of the KV Cache, further reducing storage and transmission overheads, including but not limited to: (a) Statistically analyze the numerical sequence of the KV Cache generated during the neural network inference process to establish a numerical distribution probability model; (b) Generate a dynamic arithmetic coding table based on the probability model and map the continuous numerical sequence to a variable-length binary code stream; (c) During the storage stage, use the arithmetic coding table to compress and encode the KV Cache numerical values to generate compressed data blocks; (d) During the loading stage, perform reverse arithmetic decoding on the binary code stream through a parallel decoder to restore the original numerical precision; (e) The dynamic update module periodically optimizes the probability model and the coding table according to the change of the historical numerical distribution.

5. The system according to any one of the preceding claims, characterized in that It also includes the following pseudo-RAID processing:

1. After storing the KV Cache data in a compressed format in the system memory, establish a corresponding parallel data channel group according to the number of physical channels of the memory; 2. When it is necessary to load the target KV Cache, simultaneously read multiple compressed data blocks from the memory through the parallel data channel group; 3. Perform parallel decompression processing on the read compressed data blocks (including but not limited to performing SIMD parallel decompression on multiple data blocks through the AVX-512 or AMX instruction set of the CPU, or starting multiple decompression threads through the CUDA kernel function of the GPU, with each thread processing an independent data block), and splice the decompressed data into a complete KV Cache and transfer it to the video memory.

6. The system according to any one of the preceding claims, characterized in that, The configuration method of the parallel data channel group includes:

1. Interleave the memory address space according to the number of channels; 2. Allocate an independent DMA controller for each physical memory channel; 3. Bind the corresponding relationship between the CPU core and the memory channel through the NUMA architecture.

7. The system according to any one of the preceding claims, characterized in that It also includes the following KV Cache reuse inference acceleration steps: (a) Semantically analyze the context of large language model inference requests to identify input features with potential for repeated use and generate unique context identifiers; (b) Store the pre-computed KV Cache in a compressed format in memory or RAMDISK; (c) Load the selected KV Cache from memory to the GPU video memory through a hardware-accelerated compression / decompression pipeline and perform the following operations in parallel: c1. Asynchronously decompress the cached data using tensor cores, c2. Directly map the active context segments to the attention calculation unit, c3. Reserve 5 - 10% of the video memory space for dynamically updating the highly reused cache.

8. The system according to any one of the preceding claims, characterized in that Including: (a) Deploy a distributed knowledge transfer network (KDN) on server nodes equipped with large-capacity memory to establish a shared KV cache pool; (b) Map the input context to a 128-bit hash identifier through a semantic fingerprint engine to generate a cache index key unified across models; (c) The dynamic cache allocation module optimizes the storage strategy in real time according to the following parameters: c1. The similarity matrix of the attention head structures of each model, c2. The exponentially decaying statistics of cache access frequencies, c3. The bandwidth utilization between the GPU video memory and the DDR memory, (d) Adopt a hierarchical compression strategy to differentially process KV Caches with different sharing levels: d1. Retain the FP16 precision of the highly shared blocks and directly resident in memory, d2. Apply 4-bit quantization + sparse coding compression to the medium-frequency shared blocks, d3. Transfer the low-frequency shared blocks to the NVMe SSD and establish an LRU chained index.

9. The system according to any one of the preceding claims, characterized in that , Develop a mixed-precision calculation pipeline to achieve efficient reuse: maintain a low-precision main copy such as FP16 precision in shared memory, and dynamically convert it to the native precision (FP32 / BF16) of each model when transmitted to the GPU.

10. The system according to any one of the preceding claims, characterized in that:

1. Adopt a block pre-filling technique to split a long prompt into multiple smaller blocks and allow the GPU and CPU to process in parallel to reduce inference latency; 2. Introduce hash-based prefix cache management to share the KV cache in multi-round conversations.

Images