An end-side large model inference acceleration method and device based on lossless compression
By performing bit-domain decomposition and lossless compression on large model parameters, combined with differentiated cache management and dynamic scheduling, the memory bottleneck and latency issues in large model inference on the edge side are resolved, achieving a high-efficiency improvement in model inference performance.
Patent Information
- Application Number
- CN202610493006.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-15
- Publication Date
- 2026-07-10
AI Technical Summary
The large-scale model inference process on the edge has technical problems such as high memory consumption, low runtime resource utilization and large inference latency. Existing technologies cannot effectively solve the problems of memory bottleneck and model accuracy loss.
A lossless compression-based method is used to decompose the large model parameters into bit fields, separating and compressing the exponent and fractional bits separately. Combined with differentiated cache management and dynamic scheduling mechanisms, memory management is optimized through multi-threaded parallel decompression and asynchronous execution, achieving efficient coordination between tensor loading and recovery.
While ensuring no loss of model accuracy, it significantly reduces inference latency and increases throughput, optimizes resource utilization, and improves the computing performance of edge devices.
Smart Images
Figure CN122363909A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for accelerating inference of large language models on the edge, and particularly to a method and device for accelerating inference of large language models on the edge based on lossless compression. Background Technology
[0002] In recent years, large language models have become a key technology in various industries as their performance in code generation, semantic understanding, and content creation has continuously improved. However, current large language model services mainly rely on cloud computing architecture. In this paradigm, the client needs to transmit prompts to a remote data center via a wide area network (WAN), and then the cloud returns the inference results to the client. This cloud-based interaction model has two drawbacks. First, due to the latency and instability of WAN communication, network jitter or disconnections can easily cause inference service interruptions, reducing the system's availability and robustness. Second, the long-distance network transmission of user data inevitably introduces data leakage and privacy compliance risks.
[0003] To overcome the shortcomings of cloud-based inference architectures, edge-side large-model inference technology has attracted widespread attention. This technology deploys large language models directly on user-side mobile devices or edge computing nodes, eliminating reliance on wide area network communication and achieving complete offline availability and strict privacy data isolation. However, limited by the power consumption and hardware specifications of edge devices, their computing power and physical memory capacity are far weaker than cloud-based distributed clusters, resulting in low edge-side inference efficiency. To address this issue, existing technologies often employ lossy compression techniques such as model pruning, knowledge distillation, and parameter quantization to reduce model size to fit the limited edge-side memory space. However, these lossy compression techniques inevitably alter the original weight structure and parameter distribution of the model, leading to irreversible loss of model inference accuracy. More seriously, such techniques are prone to adversarial security threats against quantized models (such as quantization attacks). That is, the model performs normally in full-precision mode, but may output malicious or unexpected content after specific quantization activation. In edge deployment scenarios lacking a centralized and controlled environment, this greatly amplifies the system's security vulnerabilities.
[0004] To mitigate accuracy loss, another existing technology attempts to employ parameter offloading or CPU-GPU hybrid inference strategies. In these approaches, some model parameters reside in GPU memory, while the remaining parameters are offloaded to CPU memory and dynamically loaded back to the GPU when needed, or the CPU is directly used as a coprocessor to work with the GPU to complete model inference. These approaches have shown some effectiveness in cloud server architectures with dedicated graphics cards and dedicated memory. However, applying these strategies directly to edge devices faces severe architectural mismatch issues. Edge computing devices are typically limited by size and power consumption, widely adopting a unified memory architecture where the CPU and GPU share the same physical memory space. Under this architecture, neither traditional offloading mechanisms nor heterogeneous hybrid inference can produce substantial physical memory expansion, failing to fundamentally reconcile the highly constrained total system memory capacity with the ever-growing volume of model parameters. Summary of the Invention
[0005] This invention addresses the technical problems of high memory consumption, low runtime resource utilization, and large inference latency in edge-side large model inference, and proposes a lossless compression-based method to accelerate edge-side large model inference.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A lossless compression-based method for accelerating large-scale inference on the edge, comprising two stages:
[0008] (1) Offline preprocessing stage:
[0009] First, the BF16 format weight tensor of the large language model deployed on the edge device is decomposed into bit fields to separate the sign bit, mantissa bit data, and exponent bit data.
[0010] Then, the exponent data is losslessly compressed to generate a compressed exponent block. The sign bit and mantissa data are packaged into a sign-mantissa block, and the exponent block, sign-mantissa block and metadata are stored together in a solid-state drive.
[0011] (2) Online reasoning stage:
[0012] Online tensor dynamic caching: Real-time awareness of memory / video memory allocation budget under the unified memory architecture on the client side, and execution of differentiated cache management based on different cache states of tensors;
[0013] Decompression Scheduling: When the network layer needs the target tensor, a directed acyclic graph (DAG) for tensor reconstruction is dynamically constructed based on the current cache state of the tensor. The DAG nodes include I / O read subtasks, CPU decompression subtasks, and tensor reconstruction subtasks, with directed edges representing dependencies between subtasks. The scheduler divides the subtasks in the DAG into I / O blocking tasks and non-blocking tasks, generating an ordered sequence of execution blocks. According to this ordered sequence, solid-state drive reading, lossless decompression of exponential bit blocks, and tensor reconstruction operations are executed asynchronously and in parallel. The reconstructed complete BF16 tensor is fed into the current network layer for model inference computation.
[0014] Furthermore, the bit field decomposition specifically includes: precisely splitting each BF16 format value into 1 sign bit, 8 exponent bits and 7 mantissa bits, and dividing the extracted exponent bit data into K independent data fragments according to a preset granularity.
[0015] Furthermore, the lossless compression algorithm includes at least one of LZ4, LZ4HC, ZSTD, or Huffman coding.
[0016] Furthermore, the differentiated cache management classifies the cache state of tensors into the following four types:
[0017] Full tensor caching: The entire tensor is stored in GPU memory;
[0018] Cache sign-mantissa bits: Only retain the sign bit and mantissa bits;
[0019] Cache compressed exponent bits: Only retain the compressed exponent bit data;
[0020] Full tensor compression: preserves both the compressed exponent and the uncompressed sign and mantissa bits.
[0021] Furthermore, when the cache pool usage exceeds the system's reserved memory threshold, the least frequently used strategy (LFU) is employed to trigger a tensor eviction operation.
[0022] Furthermore, in the DAG, the tensor reconstruction task topology corresponding to different cache states includes:
[0023] Fully cached state: No dependencies, can be used directly;
[0024] Cache only the symbol-mantissa: The following steps are required in sequence: read the compression index from the solid-state drive, decompress the CPU, and concatenate the tensor.
[0025] Only caching the compression index bits: requires reading the sign-mantissa bits from the solid-state drive and concatenating the tensor;
[0026] Full compression state: The following steps are required in sequence: read compression index bits and sign-mantissa bits from the solid-state drive, decompress the CPU, and concatenate the tensors.
[0027] Furthermore, the step of generating an ordered sequence of execution blocks includes:
[0028] Step 1: Sort the I / O blocking tasks and non-blocking tasks in descending order according to the number of tokens they need to process, generating two ordered sequences. and ;
[0029] Step 2: Initialize the execution block sequence B to be empty;
[0030] Step 3: When the sequence If not empty, repeat the following steps to construct a new execution block:
[0031] Step 3-1: According to priority, The tasks are then merged in the following order to generate an ordered priority list U of the currently schedulable tasks;
[0032] Step 3-2: Create a new execution block ,Will The highest priority task is added as the base task. ;
[0033] Step 3-3: Repeatedly select the highest priority task from U and add it. until The accumulated CPU decompression calculation time exceeds the expected I / O time;
[0034] Step 4: Construct the execution block Append to the end of block sequence B until all tasks are processed, and finally output the ordered execution block sequence B.
[0035] Furthermore, the asynchronous parallel execution includes: using a dedicated I / O thread to perform solid-state drive read operations, while simultaneously scheduling multiple CPU worker threads to perform lossless decompression operations on exponential bit blocks in parallel.
[0036] Furthermore, during tensor reconstruction, an independent CUDA stream is allocated to perform the concatenation of the sign-mantissa block and the decompressed exponent block. The sign-mantissa block and the decompressed exponent block are then concatenated in parallel by calling the CUDA Kernel function to restore the complete BF16 tensor.
[0037] An edge computing device includes a CPU and GPU with a unified memory architecture, an NVMe solid-state drive, and a memory, wherein the memory stores a computer program that, when executed by a processor, implements the method described above.
[0038] Compared with the prior art, the beneficial effects of the present invention are:
[0039] 1. This invention effectively reduces data transmission and storage volume by performing bit-domain decomposition on large model parameters and combining it with lossless compression technology to compress and store the decomposed data. During inference, the compressed data is loaded on demand and dynamically restored, optimizing runtime memory management and resource scheduling efficiency.
[0040] 2. By employing a differentiated cache management mechanism, targeted caching strategies are adopted for different bit domains, thereby effectively controlling runtime memory overhead while ensuring data integrity. Simultaneously, considering the varying resource requirements of different types of data during computation and memory access, a workflow scheduling algorithm is constructed for tensor loading and recovery processes. This achieves collaborative optimization of computation and data access, thereby reducing overall latency during inference.
[0041] 3. Experimental results show that the lossless compression-based end-side large model inference scheme proposed in this invention outperforms existing technologies in terms of system response capability and throughput performance. Under various large model architectures, this invention can reduce inter-word latency by approximately 20% to 90% and first-word latency by approximately 40% to 90%; simultaneously, it increases system throughput by approximately 1.31 to 40 times, demonstrating good versatility and significant performance improvement. Attached Figure Description
[0042] Figure 1 This is a flowchart of the end-side large model inference acceleration process based on lossless compression;
[0043] Figure 2 It is a DAG representation of the reconstruction task of tensors in different cache states;
[0044] Figure 3 This is a comparison chart of the generation latency and overhead of the system under various large model architectures;
[0045] Figure 4 This is a comparison chart of the throughput of various models under different batch sizes. Detailed Implementation
[0046] The present invention will now be described in further detail with reference to the accompanying drawings.
[0047] To address the memory bottleneck caused by the unified memory architecture of existing edge devices, as well as the model accuracy loss and security risks brought about by existing compression technologies, this invention proposes an edge-side large model inference acceleration method and device based on lossless compression. This technical solution is based on the following facts: (1) Edge computing devices generally adopt a shared physical memory architecture and lack independent video memory isolation, which leads to the failure of traditional video memory-main memory offloading scheduling. (2) The weight parameters of large language models exhibit significant entropy differences in different bit domains. Specifically, the exponent bits of the parameters have a large amount of information redundancy and are highly compressible, while the decimal / mantissa bits exhibit high entropy characteristics. (3) Modern general-purpose lossless compression algorithms can be efficiently run on CPU platforms using multi-threaded parallel processing, and their decompression throughput can be significantly higher than the I / O transmission throughput of edge devices from solid-state drives to memory.
[0048] The design and workflow of the method of this invention are as follows:
[0049] In the offline parameter decomposition and compression stage, the weight parameters of the large model are first decomposed bitwise to separate the exponent data with low entropy and high compression ratio and the decimal data with high entropy. Then, lossless compression is performed on the extracted exponent data, and the compressed exponent result and the uncompressed decimal data are stored together in the solid-state drive.
[0050] During the online tensor dynamic caching and decompression scheduling phase, the system senses the memory / GPU memory allocation budget under the current unified memory architecture in real time and dynamically adjusts the tensor caching strategy accordingly. Due to the highly limited memory capacity and I / O bandwidth of the edge devices, loading all parameters will significantly increase inference latency. Therefore, this invention presents a tensor recovery workflow scheduling method based on cache state awareness and multi-threaded decompression collaboration. When executing parameter loading, the scheduler performs optimal scheduling of the computation and I / O workflows involved in tensor recovery based on the current cache state. This allows the invention to reduce the performance loss caused by parameter unloading by utilizing the parallel decompression and replacement of tensor I / O by multi-core CPUs while ensuring zero precision loss in large model edge inference.
[0051] This invention utilizes the low entropy of the exponent and high entropy of the fractional digits in large model parameters to design a novel edge-side inference memory management and tensor loading / decompression workflow. While ensuring zero precision loss in edge-side inference for large models, this invention significantly reduces first-word latency and inter-word latency in model inference, and substantially improves the overall throughput in batch processing scenarios.
[0052] like Figure 1As shown, this invention is a lossless compression-based method for accelerating large-scale inference on the edge. Its core technology is applicable to any large language model architecture requiring parameter offloading and can be deployed on edge or terminal computing devices with a unified memory architecture. While fully preserving the original BF16 parameter semantics of the model, this method transforms the I / O-constrained inference process into a computationally intensive parallel workflow through bit-domain decomposition, lossless compression, and differentiated caching and scheduling mechanisms, thereby effectively alleviating edge-side memory bottlenecks and I / O latency issues. The method of this invention includes the following steps:
[0053] Step 1: Offline Initialization
[0054] This phase is executed once before model deployment. The execution conditions are: a hardware device with NVMe SSD storage space, and model parameters loaded in BF16 format.
[0055] Step 1.1: Bit field decomposition
[0056] Bit-level decomposition is performed on all BF16 format weight tensors to be unloaded in the large language model, precisely splitting each value in the tensor into a 1-bit sign bit, an 8-bit exponent bit, and a 7-bit mantissa bit. Further, the extracted exponent bit data is divided into K independent data fragments according to a preset granularity to facilitate subsequent parallel compression processing. This step primarily serves to provide input for subsequent compression.
[0057] Specifically, in the BF16 data format, its memory layout occupies 16 bits, including 1 sign bit, 8 exponent bits, and 7 mantissa bits. Taking the BF16 hexadecimal value 0x3F80 as an example, its memory binary sequence is 00111111 1000 0000:
[0058] (1) Sign bit (S): The highest bit is 0 (positive number).
[0059] (2) Exponent (E): The middle 8 bits are 0111 1111 (decimal 127). Subtract the offset (Bias) 127, and the actual exponent is 0.
[0060] (3) Mantissa (M): The lowest 7 bits are 000 0000.
[0061] Bit field decomposition: To perform bit field compression or custom processing, these 16 bits are decomposed into two independent uint8_t variables for storage: For the exponent bit, the 8-bit E can be directly extracted into the first uint8_t through shifting and masking. For the sign-mantissa bit, the 1-bit S and the 7-bit M can be concatenated to form the second uint8_t (S is placed in the most significant bit, and M occupies the lower 7 bits). In the reverse reconstruction stage of the tensor data, a uint16_t register variable is declared. Through bitwise OR (|) and left shift (<<) operations, the independent exponent bit uint8_t is shifted to the [7:14] interval, and the high-order bits of the sign-mantissa bit uint8_t are mapped to the 15th bit, and the low 7 bits are mapped to the [0:6] interval. This process is highly compatible with CUDA's SIMT (Single Instruction, Multiple Threads) architecture. By mapping global thread indices, each CUDA thread can independently and without branches complete the parsing and reassembly of a single BF16 scalar element.
[0062] The "preset granularity" defines the partitioning dimension and alignment strategy of the tensor in the underlying linear memory space. Assuming the input tensor contains N elements after flattening, the preset granularity parameter K indicates that the one-dimensional contiguous memory address space is divided into K data blocks using a fixed step size and equidistant partitioning. The element length of most data blocks is calculated as follows: When addressing by tensor element order, the first K-1 slices maintain a full length L. For the tail, the number of elements it carries is the remaining N - (K-1)L.
[0063] Step 1.2: Lossless Compression and Storage
[0064] The exponent bit fragments are processed using a general lossless compression algorithm (including but not limited to LZ4, LZ4HC, ZSTD, or Huffman coding) to generate compressed exponent bit blocks. Simultaneously, the sign bit and mantissa bits are directly packaged into byte-aligned sign-mantissa blocks. This step is primarily used to significantly reduce storage volume and I / O throughput.
[0065] Step 1.3: Serialization Storage
[0066] The compressed exponent tensor block, sign-mantissa tensor block, and their metadata are serialized into binary format and then unloaded and stored in the entire NVMe SSD. This step is mainly used to complete persistent storage.
[0067] Step Two: Online Real-Time Reasoning
[0068] This stage primarily involves dynamically constructing a tensor to reconstruct the topology graph and prioritizing I / O blocking tasks with decompression non-blocking tasks, leveraging multi-core CPUs to maximize the masking of storage read latency. The following steps are dynamically executed during the actual model operation.
[0069] Step 2.1: Differentiated Cache Management
[0070] The system monitors the unified memory usage budget in real time and employs a differentiated caching strategy to implement fine-grained management of tensors in different compression states. The cached states of tensors are specifically divided into the following four coexisting cases:
[0071] (1) Full Tensor Cache: The complete tensor after the forward propagation computation is completed is retained in the GPU memory.
[0072] (2) Cache sign-mantissa bits: Only retain the sign bit and mantissa bits of the tensor to avoid generating a large amount of exponent and fractional bit I / O overhead when reusing the tensor later.
[0073] (3) Cache compressed exponent bits: Only the compressed exponent bit data is kept in memory.
[0074] (4) Full tensor compression: retains the compressed exponent bits as well as the uncompressed sign and decimal bits.
[0075] During operation, when the cache pool usage exceeds the system's reserved memory threshold, the system triggers a tensor eviction operation using the Least-Frequently Used (LFU) strategy.
[0076] By implementing fine-grained state management for tensors of different bit fields, the optimal balance between computation and memory access is achieved under limited memory conditions.
[0077] The system utilizes a cache pool partitioning and dynamic allocation mechanism to schedule and transition between different cache states. Specifically, during the initialization phase, the system allocates corresponding memory space budgets to four cache states based on user-preset cache pool ratios (determined by hardware resource conditions and performance requirements). These four cache states include: full tensor cache, full tensor compressed cache, sign-mantissa cache, and compressed exponential cache. During operation, the system maintains tensor access frequency statistics and sorts all tensors based on their access frequency. According to the preset cache pool ratios, the sorted tensors are allocated to different cache pools sequentially: the tensor with the highest access frequency is allocated to the full tensor cache, followed by the full tensor compressed cache, then the sign-mantissa cache, and the tensor with the lowest access frequency is allocated to the compressed exponential cache. When the tensor access frequency changes or a new tensor is added, the system re-executes the above allocation strategy based on the updated frequency sort, thereby achieving dynamic migration and transition of tensors between different cache states.
[0078] The system's "memory usage budget" is pre-set by the user based on the resource constraints of the target hardware platform. During operation, the system dynamically calculates the current memory usage by statistically analyzing the metadata of cached tensors. Specifically, the system maintains a tensor metadata table to record the storage space occupied by each tensor in its current cached state. When new cache allocations or state transitions occur, the system updates this table in real time and accumulates the current total memory overhead.
[0079] The system employs a frequency recording mechanism based on hash tables. Specifically, each tensor maintains an access counter, and a hash table structure is used to map tensor identifiers to access frequencies. Whenever a tensor is accessed, the corresponding counter is atomically incremented. During scheduling or cache reallocation phases, the system can sort tensors according to their access frequencies in the hash table, thereby implementing LFU-based cache decisions.
[0080] Step 2.2: DAG Construction and Workflow Scheduling
[0081] During inference, when a specific tensor is needed at a certain layer of the network, the scheduler dynamically constructs a Directed Acyclic Graph (DAG) describing the tensor reconstruction operation based on the tensor's current cache state. In this DAG, nodes represent the required computational or I / O subtasks (including SSD reads, CPU decompression, and tensor reconstruction operations), and directed edges represent logical dependencies between subtasks. The topology of each reconstruction task varies depending on the cache state, as shown below. Figure 2 As shown in the diagram, in the DAG representation, all edges represent data dependencies, and all DAG structures do not need to be generated according to any certain rule; they are uniquely determined based on the corresponding cache state.
[0082] Step 2.3: Workflow Scheduling
[0083] The scheduler divides all subtasks into Class I tasks (I / O blocking tasks that require loading a symbol-mantissa tensor block) and Class II tasks (non-blocking tasks with a cached symbol-mantissa tensor block). The scheduling algorithm constructs execution blocks according to the following process:
[0084] (1) First, sort the tasks of type I and type II in descending order according to the number of tokens they need to process, and generate two ordered sequences. and .
[0085] (2) The set of execution blocks B is initialized to be empty.
[0086] (3) When the sequence If not empty, repeat the following steps to construct a new execution block:
[0087] (3-1) According to priority, The tasks are then merged in the next order to generate an ordered priority list U of currently schedulable tasks.
[0088] (3-2) Create a new empty block ,Will The highest priority task is added as the base task.
[0089] (3-3) Enter the sub-loop until the block is reached. The accumulated expected CPU decompression calculation time exceeds the expected I / O time (the expected CPU decompression time and I / O time can be obtained through offline benchmark testing and recorded in the configuration file maintained by the system. During runtime, the system obtains the expected CPU decompression time and I / O time by reading the configuration file):
[0090] (3-3-1) Select the highest priority task to be scheduled from list U. ;
[0091] (3-3-2) Task Add to The tail;
[0092] (3-3-3) Remove the task from U If U is empty, then the sub-loop ends.
[0093] a. The completed execution block Add to the end of set B.
[0094] (4) Repeat step (3) until all tasks are completed, and finally output the ordered sequence of execution blocks. .
[0095] Step 2.4: Asynchronous parallel execution of subtasks
[0096] After submitting tasks to the thread pool and GPU, the system strictly follows the task scheduling order in the ordered execution block sequence B. The system uses a dedicated I / O thread to handle all solid-state drive read operations, while simultaneously scheduling multiple CPU worker threads to perform lossless decompression operations on exponential bit tensor blocks in parallel.
[0097] Step 2.5: Tensor Reconstruction and Model Inference
[0098] After all the preceding subtasks of a specific tensor reconstruction operation are completed, the system allocates an independent CUDA Stream to handle the subsequent tensor reconstruction work. Specifically, the sign-mantissa tensor block obtained using I / O operations and the decompressed exponent tensor block are concatenated in parallel using CUDA Kernel functions to restore the complete BF16 tensor, which is then directly fed back to the current network layer for computation.
[0099] The system divides data processing into two parallel execution phases: I / O operations and CPU decompression operations. The I / O thread is responsible for reading compressed data from external storage, while the computation thread is responsible for performing decompression and memory copy operations. This invention employs a task-granularity-based asynchronous synchronization mechanism. Specifically, each task is broken down into multiple sub-operations. Upon completion of each sub-operation, a completion marker is updated via a shared task state object. The task state object maintains a completion counter, which is updated atomically when any sub-operation ends. When the counter meets a preset completion condition, a task completion callback is triggered. In the task completion callback, the system moves the corresponding task from the scheduling queue to the ready queue and notifies the waiting thread via a condition variable, thereby achieving cross-thread synchronization and state transfer. After the CPU completes all pre-order operations, the system transfers the data to the GPU via the CUDA Runtime API and triggers a tensor recovery operation. The recovery operation is executed in a separate CUDA Stream. `cudaEventRecord` is used to record the completion event in the CUDA stream for tensor recovery, and `cudaStreamWaitEvent` is called in the CUDA stream where forward inference resides, keeping it in a waiting state until the event is completed. This mechanism ensures that tensor splicing and subsequent computations occur strictly after decompression and data transfer are complete, thus achieving correct synchronization and efficient pipelined execution between CPU decompression and GPU computation. In abnormal situations, the system attempts to retry failed operations.
[0100] Comparison of generation latency overhead of systems under various large model architectures, such as Figure 3 As shown, the throughput of various models under different batch sizes is compared as follows: Figure 4 As shown. The comparison methods used in this embodiment (including Accelerate, DeepSpeed, and MoE-Infinity, all three being open-source high-performance inference frameworks) are implemented using an NVIDIA Jetson AGXOrin (12-core ARM CPU, 2048-core Ampere GPU, 32GB unified memory) equipped with a Samsung 970 EVO 512G SSD, providing a read throughput of 3.5GB / s. The DeepSeekV2-Lite 16B, Qwen1.5-MoE 14B, and SwitchTransformers-Large-128 26B models were used for testing. Prompt words obtained through random sampling from ShareGPT were used for testing. All comparison methods sampled the same set of prompt words to ensure fairness. Experimental results show that under various large model architectures, this invention can reduce inter-word latency by approximately 20%–90% and first-word latency by approximately 40%–90%; simultaneously, the system throughput is increased by approximately 1.31 times to 40 times, demonstrating good versatility and significant performance improvement. It is evident that the lossless compression-based end-side large model inference scheme proposed in this invention outperforms existing technologies in terms of system response capability and throughput performance.
[0101] The above embodiments are merely typical implementations of the present invention and are not intended to limit the present invention. All equivalent substitutions or improvements made within the scope of the claims of the present invention are within the protection scope of the present invention.
Claims
1. A method for accelerating large-scale end-side inference based on lossless compression, characterized in that, It includes two stages: (1) Offline preprocessing stage: First, the BF16 format weight tensor of the large language model deployed on the edge device is decomposed into bit fields to separate the sign bit, mantissa bit data, and exponent bit data. Then, the exponent data is losslessly compressed to generate a compressed exponent block. The sign bit and mantissa data are packaged into a sign-mantissa block, and the exponent block, sign-mantissa block and metadata are stored together in a solid-state drive. (2) Online reasoning stage: Online tensor dynamic caching: Real-time awareness of memory / video memory allocation budget under the unified memory architecture on the client side, and execution of differentiated cache management based on different cache states of tensors; Decompression Scheduling: When the network layer needs the target tensor, a directed acyclic graph (DAG) for tensor reconstruction is dynamically constructed based on the current cache state of the tensor. The DAG nodes include I / O read subtasks, CPU decompression subtasks, and tensor reconstruction subtasks, with directed edges representing dependencies between subtasks. The scheduler divides the subtasks in the DAG into I / O blocking tasks and non-blocking tasks, generating an ordered sequence of execution blocks. According to this ordered sequence, solid-state drive reading, lossless decompression of exponential bit blocks, and tensor reconstruction operations are executed asynchronously and in parallel. The reconstructed complete BF16 tensor is fed into the current network layer for model inference computation.
2. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, The bit field decomposition specifically includes: precisely splitting each BF16 format value into 1 sign bit, 8 exponent bits, and 7 mantissa bits, and dividing the extracted exponent bit data into K independent data fragments according to a preset granularity.
3. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, The lossless compression algorithm used includes, but is not limited to, LZ4, LZ4HC, ZSTD, and Huffman.
4. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, The differentiated cache management classifies the cache state of tensors into the following four types: Full tensor caching: The entire tensor is stored in GPU memory; Cache sign-mantissa bits: Only retain the sign bit and mantissa bits; Cache compressed exponent bits: Only retain the compressed exponent bit data; Full tensor compression: preserves both the compressed exponent and the uncompressed sign and mantissa bits.
5. The method for accelerating large-scale end-side inference based on lossless compression according to claim 4, characterized in that, When the cache pool usage exceeds the system's reserved memory threshold, the least frequently used strategy (LFU) is used to trigger a tensor eviction operation.
6. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, In the DAG, the tensor reconstruction task topology corresponding to different cache states includes: Fully cached state: No dependencies, can be used directly; Cache only the symbol-mantissa: The following steps are required in sequence: read the compression index from the solid-state drive, decompress the CPU, and concatenate the tensor. Only caching the compression index bits: requires reading the sign-mantissa bits from the solid-state drive and concatenating the tensor; Full compression state: The following steps are required in sequence: read compression index bits and sign-mantissa bits from the solid-state drive, decompress the CPU, and concatenate the tensors.
7. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, The step of generating an ordered sequence of execution blocks includes: Step 1: Sort the I / O blocking tasks and non-blocking tasks in descending order according to the number of tokens they need to process, generating two ordered sequences. and ; Step 2: Initialize the execution block sequence B to be empty; Step 3: When the sequence If not empty, repeat the following steps to construct a new execution block: Step 3-1: According to priority, The tasks are then merged in the following order to generate an ordered priority list U of the currently schedulable tasks; Step 3-2: Create a new execution block ,Will The highest priority task is added as the base task. ; Step 3-3: Repeatedly select the highest priority task from U and add it. until The accumulated CPU decompression calculation time exceeds the expected I / O time; Step 4: Construct the execution block Append to the end of block sequence B until all tasks are processed, and finally output the ordered execution block sequence B.
8. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, The asynchronous parallel execution includes: using a dedicated I / O thread to perform solid-state drive read operations, while simultaneously scheduling multiple CPU worker threads to perform lossless decompression operations on exponential bit blocks in parallel.
9. The method for accelerating large-scale end-side inference based on lossless compression according to claim 1, characterized in that, During tensor reconstruction, an independent CUDA stream is allocated to perform the concatenation of the sign-mantissa block and the decompressed exponent block. The CUDA Kernel function is called in parallel to reconstruct the complete BF16 tensor by concatenating the sign-mantissa block and the decompressed exponent block.
10. An edge computing device, comprising a CPU and GPU with a unified memory architecture, an NVMe solid-state drive, and a memory, characterized in that, The memory stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.