Large language model efficient reasoning method and framework for resource-constrained end-side equipment

Abstract

The invention discloses a large language model efficient reasoning method and framework for resource-constrained end-side equipment, and belongs to the technical field of Web operating systems. According to the framework, through cooperative work of a task scheduling layer, an inference engine layer and a resource treatment layer, the problem of adaptation of tasks and models is solved; the reasoning engine layer improves the reasoning throughput while reducing the occupancy peak value of a GPU video memory through draft model unloading, double-batch staggered assembly line and asynchronous loading; the resource governance layer monitors CPU, GPU and memory resources in real time, provides data support for the dynamic adjustment strategy of the reasoning engine layer, and avoids resource overload or waste. According to the framework, high video memory efficiency is realized on a data set, and practical framework support is provided for LLM deployment of domestic end side software and hardware ecology.

Description

Technical Field

[0001] This application relates to the field of Web operating system technology, and in particular to an efficient reasoning method and framework for large language models for resource-constrained end devices. Background Technology

[0002] With the advent of the AIPC era, the demand for localized inference of Large Language Models (LLM) on edge devices (smartphones, embedded development boards, etc.) has surged. It can avoid the privacy risks of cloud transmission and reduce inference latency to the millisecond level, which is the core direction for achieving efficient intelligent interaction. However, edge devices generally face multiple limitations in computing power, storage, and hardware architecture adaptation. Specifically, the limited resources of edge devices are a prominent issue. Their computing power is only 1 / 10 to 1 / 100 of that of cloud servers, with memory ranging from 2 to 16GB and storage from 32 to 256GB, while the original size of mainstream LLMs exceeds 20GB and has high temporary memory usage. At the same time, the devices cover multiple architectures such as ARM, x86, and RISC-V, with significant differences in instruction set support, which significantly increases the difficulty of adapting LLM inference. Existing solutions, such as lightweight models lacking architectural specificity, resource scheduling using a fixed allocation mode with delayed response, insufficient coordination between inference execution and result reuse, and lack of verification and reprocessing mechanisms for hardware compatibility adaptation, cannot solve the problems of "difficult adaptation, low utilization, poor efficiency, and weak compatibility".

[0003] Therefore, there is an urgent need for an efficient LLM inference solution that adapts to edge-side characteristics. Summary of the Invention

[0004] This application provides an efficient reasoning method and framework for large language models on resource-constrained edge devices. The technical solution is as follows: On the one hand, an efficient reasoning framework for large language models is provided for resource-constrained edge devices. The framework includes: The task scheduling layer is configured with an inference request receiving unit and an inference task decomposition unit, which are used to receive LLM inference requests from the end side and decompose the inference task into sub-tasks adapted for dual-model collaborative processing. The inference engine layer includes a dual-model collaborative inference unit, a dual-batch interleaved pipeline unit, a draft model unloading unit, and an asynchronous loading unit. The dual-model collaborative inference unit is used to perform draft model prediction and target model verification. The dual-batch interleaved pipeline unit is used to interleave two sequences to improve inference efficiency. The draft model unloading unit is used to unload the multi-layer perceptron (MLP) components and key-value cache (KVcache) of the draft model to CPU memory. The asynchronous loading unit is used to prefetch model components during the calculation process. The resource governance layer includes a computing resource monitoring unit and a memory resource monitoring unit, which are used to monitor the CPU utilization, GPU memory usage and memory usage of edge devices in real time, and communicate with the inference engine layer to dynamically adjust the allocation of inference resources.

[0005] Optionally, in the draft model unloading unit of the inference engine layer, the draft model is a Mistral-type non-mixture of experts (MoE) architecture model. During the unloading process, the LLM attention calculation task is delegated to the CPU for execution, and the current layer activation value (Query) is transferred from the GPU to the CPU. After the calculation is completed, the result is returned to the GPU.

[0006] Optionally, the inference engine layer also includes a fine-grained component loading unit for loading each MoE layer of the target MoE architecture model. The fine-grained component loading unit uses multi-threading technology to prefetch gating networks and expert components in parallel. The target MoE architecture model contains 8 expert components in each MoE layer. The multi-threading technology starts 9 threads to load 1 gating network and 8 expert components respectively. The expert components refer to independent model functional modules used to perform specific inference computing tasks.

[0007] Optionally, the inference engine layer also includes a Load-to-GPU Pinning unit, which is used to open a physically contiguous paged memory region in the host memory and persistently reside the weights of the MLP component and the MoE component in this region. The locked page memory region is 10.83GB in size and supports zero-copy pass-through transmission via tensor.to(device,non_blocking=True) for direct memory access (DMA). When the Load-to-GPUPinning unit performs memory locking, it includes: A 10.83GB physical contiguous region is allocated in the host memory as a page lock buffer. The weights of the MLP and MoE components are stored in this region. During inference, DMA achieves zero-copy transfer of component weights by directly accessing the page lock buffer, eliminating the overhead of paging table lookup and double copy.

[0008] Optionally, the monitoring frequency of the computing resource monitoring unit and the memory resource monitoring unit in the resource governance layer is real-time; wherein, The monitored data is used to obtain resource usage information, including CPU core utilization, peak GPU memory usage, and memory usage of edge devices.

[0009] On the other hand, an efficient inference method for large language models for resource-constrained edge devices is provided, applicable to the LLM inference framework described above. The method includes: The task scheduling layer of the framework receives the LLM inference request from the end side, and the inference task decomposition unit decomposes the inference task into sub-tasks that are adapted for dual-model collaborative processing. The dual-batch interleaved pipeline unit of the inference engine layer controls the dual-model collaborative inference unit to perform draft model prediction and target model verification in a sequential interleaved manner. At the same time, the draft model unloading unit unloads the MLP components and KVcache of the draft model to the CPU memory, and the asynchronous loading unit prefetches the model components required for inference. The resource governance layer monitors the CPU usage, GPU memory usage, and memory usage of the edge devices in real time, and feeds the monitoring data back to the inference engine layer to dynamically adjust the draft model unloading, asynchronous loading, and component loading strategies.

[0010] Optionally, when the dual-model collaborative reasoning unit performs draft model prediction and target model verification, it adopts a dual-batch interleaving process. The double-batch interleaving process includes: For two sequences to be inferred, the draft model first processes the prediction subtask of sequence 1, while the target model simultaneously verifies the prediction result of sequence 2, and then alternately processes the prediction of sequence 2 and the verification of sequence 1.

[0011] Optionally, the draft model unloading unit is used to asynchronously prefetch the weights of the current layer MLP components to GPU memory while the CPU performs attention calculation, so as to achieve the overlap of calculation and weight transmission. The attention calculation only transmits the current layer Query activation value, where the data volume is O(b*s*d), b is the batch size, s is the sequence length, and d is the hidden layer dimension.

[0012] Optionally, when the inference engine layer performs fine-grained component loading, the method further includes: For each MoE layer of the target MoE architecture model, nine independent threads are used for parallel loading, with one thread loading the gated network component and eight threads loading eight expert components respectively. Once each component is loaded, its corresponding computation begins.

[0013] This application discloses an efficient inference method and framework for large language models on resource-constrained edge devices, belonging to the field of Web operating system technology. The framework achieves the core goals of "maximizing GPU memory efficiency and maintaining accuracy" for LLM inference on resource-constrained edge devices through the collaborative work of a task scheduling layer, an inference engine layer, and a resource governance layer, solving the task-model adaptation problem. The inference engine layer improves inference throughput while reducing peak GPU memory usage through draft model unloading, dual-batch interleaved pipelines, and asynchronous loading. The resource governance layer monitors CPU, GPU, and memory resources in real time, providing data support for the dynamic adjustment strategy of the inference engine layer and avoiding resource overload or waste. Ultimately, the framework achieves high GPU memory efficiency on the HumanEval, Summeval, and Samsum datasets, and maintains lossless inference accuracy because it does not employ compression strategies such as quantization or pruning. It effectively adapts to the localization, low latency, and high privacy AI requirements of edge hardware and software, providing practical architectural support for LLM deployment in the domestic edge hardware and software ecosystem. Attached Figure Description

[0014] Figure 1 A comparison chart of video memory efficiency is shown; Figure 2 An overall architecture diagram of E-SpecOffload is shown; Figure 3 A schematic diagram of an E-SpecOffload optimization execution mechanism is shown. Figure 4 This figure shows a comparison of video memory efficiency for the code generation dataset HumanEval; Figure 5 This figure shows a comparison of memory efficiency for the text summarization dataset Summeval. Figure 6 This figure shows a comparison of memory efficiency for the dialogue summarization dataset Samsum. Figure 7 A schematic diagram comparing the dynamic memory usage of E-SpecOffload and SpecOffload during inference is shown. Figure 8 This diagram illustrates a comparison of memory usage across different frameworks. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0016] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0017] First, through Figure 1 Here's a brief overview of the current research status, where GenerationLength represents the number of tokens generated during model inference. The vertical axis represents memory efficiency.

[0018] Dynamic Offloading is a common and fundamental method in optimizing large language model inference on edge devices. Its core idea is to treat the model as a whole, dynamically removing unnecessary parts (such as certain layers, weights, or KV cache) from GPU memory and storing them in CPU memory or external storage, then loading them on demand. This method maintains the accuracy of the original model while offering high flexibility, dynamically adjusting data residency strategies based on available GPU memory, thus becoming a highly promising solution for deploying large language models (LLMs) on edge devices. Representative works include Fiddler, which effectively solves the problem of running large MoE models in resource-constrained environments by intelligently scheduling CPU and GPU resources and dynamically selecting the optimal execution strategy, improving inference speed while maintaining accuracy. SpecOffload further embeds speculative decoding (SD, often used in resource-rich inference acceleration) into the offloading mechanism, significantly improving inference throughput.

[0019] However, despite some progress made by existing methods, resource-constrained edge environments still face a fundamental challenge: the inherent trade-off between throughput and memory efficiency. SpecOffload excels in throughput, but its memory efficiency (i.e., throughput supported per unit of video memory) in edge environments is relatively low (e.g., ...). Figure 1 The memory efficiency comparison chart shown is lower than that of Fiddler. The main reason for this phenomenon is that it has a high fixed memory usage during inference, which limits memory efficiency and thus seriously affects its practical value for edge deployment. Therefore, our primary goal is to reduce SpecOffload memory usage while avoiding the degradation of system performance (such as throughput) caused by the additional data transfer overhead introduced by memory compression, thereby maximizing memory efficiency.

[0020] Therefore, we propose memory efficiency (η) as the core optimization metric for edge scenarios. Traditional optimization metrics (such as single throughput or memory usage) are difficult to effectively measure the overall performance of LLM inference in resource-constrained environments, leading to existing solutions tending to sacrifice the balance between resource utilization and speed. Therefore, to more comprehensively and accurately reflect the actual performance of the LLM inference framework in edge deployment, we propose memory efficiency (η) as the core optimization metric for edge scenarios, defined as follows: η = T / M, where T represents throughput, i.e., the number of tokens generated by the model per unit time; M represents the peak memory usage. This metric directly reflects the inference throughput that can be obtained per unit of memory resources, and can intuitively measure the "input-output ratio" of memory usage.

[0021] Improving memory efficiency according to the formula can be approached from two directions: reducing peak memory usage (M) or increasing throughput (T). However, in actual edge deployments, there is often a trade-off between these two. Memory compression may introduce computational overhead, while throughput optimization may increase memory pressure. Memory efficiency (η) provides a unified quantitative criterion for LLM inference on edge devices, which can effectively guide a better trade-off between memory and throughput, and help achieve a better performance balance under practical constraints.

[0022] Our optimizations aim to reduce GPU memory usage while increasing throughput, thereby achieving overall memory efficiency (η). To this end, we propose E-SpecOffload (a framework for efficient inference of large language models for resource-constrained edge devices) based on SpecOffload, reducing SpecOffload's GPU memory footprint by offloading draft models. However, this offloading inevitably reduces throughput. To mitigate this issue, we parallelize the loading of target model components and pin frequently accessed modules to CPU memory, thereby minimizing data transfer latency. Ultimately, E-SpecOffload achieves an effective balance between throughput and memory usage. The architecture of E-SpecOffload is as follows: Figure 2 As shown, after receiving an inference request, the inference task is broken down. The broken-down task enters the core dual-model collaborative inference stage, where the draft model predicts and the target model verifies. Notably, this stage is organized and executed using a dual-batch interleaved pipeline, with the draft model and target model interleaving the two sequences to improve efficiency. During this process, the system unloads the draft model and uses an asynchronous loading strategy. Simultaneously, the background continuously monitors computation and memory resource usage in real time throughout the entire process. The E-SpecOffload framework will be further explained below through various embodiments.

[0023] Example 1 To reiterate, in each embodiment, E-SpecOffload refers to an efficient inference framework for large language models oriented towards resource-constrained edge devices.

[0024] like Figure 2 As shown, an efficient reasoning framework for large language models for resource-constrained edge devices is provided. The framework includes the following components.

[0025] First, the task scheduling layer is equipped with an inference request receiving unit and an inference task decomposition unit, which are used to receive LLM inference requests from the end side and decompose the inference task into sub-tasks adapted for dual-model collaborative processing.

[0026] In this embodiment, the edge device adopts a hardware configuration (GPU is A100-SXM4, VRAM is 40GB, CPU is Xeon Platinum 8378A, DRAM is 1TB) and runs the E-SpecOffload framework built on HuggingFaceTransformers v4.47.1. The inference request receiving unit of the task scheduling layer receives the "HumanEval dataset code generation inference request" initiated by the user (this dataset contains 164 programming problems), and the request content is "generate quicksort function code in Python".

[0027] The reasoning task decomposition unit breaks down the reasoning task into two sub-tasks based on the request type and the framework dual-model collaborative logic.

[0028] Subtask 1 is the "Draft Model Prediction Subtask" (generating a preliminary code snippet of 8 tokens from the Mistral-7B draft model). In this example, Mistral specifically refers to the Mistral series of open-source large language models, and the Mistral-7B draft model is a 7 billion parameter open-source large language model in the Mistral series.

[0029] Subtask 2 is the "Target Model Validation Subtask" (the Mixtral-8x7B target model (46.7B parameters, bfloat16 precision) verifies the correctness of the fragments generated by the draft model; if correct, it continues generation; if incorrect, it is corrected). After decomposition, the two subtasks are passed to the inference engine layer through the framework's internal data interface, ensuring that the two models can be processed collaboratively in sequence.

[0030] Second, the inference engine layer includes a dual-model collaborative inference unit, a dual-batch interleaved pipeline unit, a draft model unloading unit, and an asynchronous loading unit. The dual-model collaborative inference unit is used to perform draft model prediction and target model verification. The dual-batch interleaved pipeline unit is used to interleave two sequences to improve inference efficiency. The draft model unloading unit is used to unload the MLP components and KVcache of the draft model to CPU memory. The asynchronous loading unit is used to prefetch model components during the calculation process, where MLP is a multilayer perceptron and KVcache is a key-value cache.

[0031] Based on the above hardware configuration, each unit of the inference engine layer works collaboratively in this embodiment.

[0032] Specifically, the dual-model collaborative reasoning unit loads the Mistral-7B draft model and the Mixtral-8x7B target model, where the target model is a MoE architecture, and each MoE layer contains 8 expert components.

[0033] The dual-batch interleaved pipeline unit selects two sequences to be inferred (both from code generation requests in the HumanEval dataset: sequence 1 is "generate quicksort function", and sequence 2 is "generate binary search function") and processes them in an interleaved manner.

[0034] In the first time slice, the draft model processes the prediction subtask of Sequence 1 (generating code snippets of 8 tokens), while the target model processes the verification subtask of Sequence 2 (verifying the Sequence 2 snippet generated by the previous draft model).

[0035] In the second time slice, the draft model switches to prediction of sequence 2, while the target model switches to verification of sequence 1, achieving parallel processing of the two sequences.

[0036] The draft model unloading unit performs the unloading operation. Specifically, it migrates all MLP components (weight file size approximately 2.8GB) and KVcache (for inference tasks with a sequence length of 512, KVcache data size approximately O(32*512*4096), where b=32 is the batch size, s=512 is the sequence length, and d=4096 is the hidden layer dimension) of the Mistral-7B draft model to CPU memory (DRAM corresponding to Xeon Platinum 8378A). Only when speculative calculations are needed, the Query activation value of the current layer (data size O(32*4096)=131072 dimensions) is transferred from the GPU to the CPU.

[0037] While the CPU performs attention calculations (handling the matching of queries and KVcache), the asynchronous loading unit prefetches the weights of the current layer MLP components through multi-threading (prefetching from CPU memory to A100 GPU memory). The prefetching action is performed synchronously with the attention calculations, with no waiting delay.

[0038] Third, the resource governance layer includes a computing resource monitoring unit and a memory resource monitoring unit, which are used to monitor the CPU utilization, GPU memory usage and memory usage of the edge devices in real time, and communicate with the inference engine layer to dynamically adjust the allocation of inference resources.

[0039] In this embodiment, the computing resource monitoring unit and memory resource monitoring unit of the resource governance layer collect hardware data in real time for the above-mentioned code generation inference task.

[0040] The computing resource monitoring unit collects CPU (Xeon Platinum 8378A) utilization (stable at 65%-70% during inference) and GPU (A100-SXM4) memory usage (from...) every 100ms. Figure 7 According to available information, the memory usage related to the target model (corresponding to E-Traget) is approximately 3.75GB, the memory usage related to the draft model (corresponding to E-Draft) is approximately 4.5GB, and the total memory usage (corresponding to E-Total) is approximately 7.5~8GB. The memory resource monitoring unit synchronously collects the DRAM memory usage rate (after the draft model MLP component and key-value cache KVcache are unloaded, the memory usage increases from 120GB to 210GB, which is still about 21% of the total 1TB memory and does not exceed the memory carrying capacity threshold of the edge device).

[0041] In summary, such as Figure 3 The diagram illustrates the optimization mechanism of E-SpecOffload. It showcases the three core optimization mechanisms of the E-SpecOffload framework. On the left, the draft model side utilizes a "draft model offloading" technique, offloading the KVcache and MLP components of the draft model to CPU memory. Only the activation values ​​are transferred from the GPU to the CPU, where the CPU performs attention calculations. The results are then returned to the GPU, significantly reducing GPU memory usage (e.g., in the Mistral-7B example, the draft model only transfers O(b*d) query activation values). On the right, the target model side employs a "fine-grained component loading" technique for the MoE architecture (including the gate network and multiple expert components). This involves multi-threading to prefetch the gate and expert components in parallel, and the computation is executed immediately after component loading (i.e.,...). Figure 3 The framework employs a module-level immediate execution mechanism, eliminating the need to wait for all components to load (e.g., in the Mixtral-8x7B MoE layer of the example, 9 threads load 1 gating component and 8 expert components respectively; computation begins as soon as the gating component is loaded, corresponding to fine-grained loading). The bottom-level Load-to-GPU Pinning mechanism allocates a physically contiguous paged memory region in host memory, using DMA (Direct Memory Access) to achieve zero-copy transfer of model components, eliminating the overhead of paging table lookups and double copying (e.g., in the example, 10.83GB of paged memory is allocated using mlock(), reducing component transfer time from 15ms to 3ms, corresponding to memory locking technology). With these three elements working together, the framework achieves "reduced GPU memory usage + increased inference throughput" for LLM inference in resource-constrained scenarios on the client side.

[0042] Load-to-GPUPinning is a GPU memory locking technique, an optimization technique that fixes the weights of model components in a physically contiguous memory region.

[0043] In summary, the embodiments of this application have the following technical effects.

[0044] The task scheduling layer solves the adaptation problem between inference tasks and dual-model collaborative logic by "receiving specific inference requests and breaking them down into dual-model sub-tasks". It transforms abstract edge-side inference requirements (such as code generation and text summarization) into specific sub-tasks that are predictable in the draft model and verifiable in the target model, ensuring that the two models can work collaboratively in sequence and avoiding inference process chaos caused by unclear task decomposition. Simultaneously, by designing sub-tasks in conjunction with datasets (HumanEval, Summeval, Samsum), the framework can directly adapt to common edge-side LLM inference scenarios, improving practicality.

[0045] The inference engine layer achieves two core effects through the collaborative work of units such as dual-model collaboration, dual-batch interleaving pipeline, draft model offloading, asynchronous loading, and fine-grained component loading: First, it significantly reduces GPU memory usage by offloading the MLP components and KVcache of the draft model to CPU memory, reducing GPU memory usage by 50% compared to SpecOffload (where the draft model resides in GPU memory); Second, it greatly improves inference throughput. Dual-batch interleaving shortens the inference time of the two sequences by 40% compared to serial processing, and asynchronous loading and fine-grained component loading eliminate I / O latency. Based on the offloading of the draft model, it optimizes performance stability and ultimately achieves a balance between "low memory usage + high throughput".

[0046] The resource governance layer resolves resource conflicts in resource-constrained environments by monitoring CPU, GPU, and memory resources in real time and dynamically adjusting strategies. This prevents inference crashes caused by GPU memory overload or computational lag caused by insufficient CPU resources. Meanwhile, real-time feedback of monitoring data enables the inference engine layer to accurately adjust unloading and loading strategies to maximize resource utilization efficiency. For example, it maintains loading speed when GPU memory is sufficient and releases unnecessary resources when the CPU is overloaded, ensuring stable operation of inference tasks.

[0047] Example 2 This embodiment is based on the framework of Embodiment 1. The inference task is text summarization generation from the Summeval dataset. In the draft model offloading unit of the inference engine layer, the draft model is Mistral-7B, a non-MoE architecture. The draft model is a Mistral-type non-MoE architecture model. During the offloading process, the LLM attention calculation task is delegated to the CPU, and the current layer activation value (Query) is transferred from the GPU to the CPU. After the calculation is completed, the result is returned to the GPU.

[0048] In one possible implementation, when the draft model unloading unit performs unloading, it first migrates the Mistral-7B's MLP components (containing two fully connected layers with weight dimensions of 4096×16384 and 16384×4096 respectively) and KVcache (for the summary generation task, the sequence length is 1024, the KVcache data size is O(16*1024*4096), and b=16 is the batch size) to CPU memory. When attention calculation is required, only the current layer's query activation value (data size O(16*4096) = 65536 dimensions) is transferred from the GPU to the CPU via the PCIeGen4x16 interface. After the CPU completes the attention score calculation (the dot product of the query and KVcache), it returns the calculation result (also 65536-dimensional attention weights) to the GPU to continue subsequent inference steps. Throughout the process, KVcache remains resident in CPU memory, avoiding I / O overhead caused by repeated migration.

[0049] Optionally, the inference engine layer also includes a fine-grained component loading unit for each MoE layer of the target MoE architecture model. The fine-grained component loading unit prefetches gated networks and expert components in parallel using multi-threading technology.

[0050] In the target MoE architecture model, each MoE layer contains 8 expert components, and multi-threading technology enables 9 threads to load 1 gating network and 8 expert components respectively.

[0051] In one possible implementation, the fine-grained component loading unit of the inference engine layer performs a loading operation on the 15th MoE layer (out of 32 MoE layers) of the Mixtral-8x7B.

[0052] Nine independent threads are used. Thread 1 is responsible for loading the gated network component of the MoE layer (the weight file size is about 0.5GB). Threads 2-9 are responsible for loading eight expert components (each expert component weight size is about 1.2GB). The gated network component of thread 1 is loaded first (taking about 20ms) and immediately starts the gating calculation (weighting the input features) without waiting for the expert components of threads 2-9 to load. When the expert component 1 of thread 2 is loaded (taking about 35ms), it immediately receives the feature data allocated by the gated network and starts the calculation. The subsequent expert components start the calculation in the order of loading completion until all expert components have completed the calculation and the results are summarized. Expert components refer to independent model functional modules used to perform specific inference calculation tasks.

[0053] Optionally, the inference engine layer also includes a Load-to-GPU Pinning unit, which is used to allocate a physically contiguous paged memory region in the host memory, and persistently reside the weights of the MLP component and the MoE component in this region. The paged memory region is 10.83GB in size and supports zero-copy pass-through through tensor.to(device,non_blocking=True) for direct DMA access.

[0054] When the Load-to-GPU Pinning unit performs memory locking, it allocates a 10.83GB physical contiguous region in the host memory as a page lock buffer, stores the weights of MLP and MoE components in this region, and achieves zero-copy transfer of component weights by directly accessing the page lock buffer during inference, eliminating the overhead of page table lookup and double copy.

[0055] In other words, the Load-to-GPU Pinning unit allocates a 10.83GB physical contiguous page memory region in the host DRAM memory using the Linux system's `mlock()` function. This region is divided into two sub-partitions. 5GB is used to store the weights of the Mixtral-8x7B MLP components (a total of 32 Transformer layers, with each MLP component having a weight of approximately 0.15GB), and 5.83GB is used to store the weights of the MoE components (32 MoE layers, with each MoE layer containing 8 expert components, for a total weight of approximately 5.5GB).

[0056] After migrating the component weights from ordinary paged memory to paged memory, zero-copy pass-through is achieved through PyTorch's `tensor.to('cuda',non_blocking=True)` interface. The A100 GPU's DMA controller directly accesses the weight data in the paged memory without having to copy the data from paged memory to fixed memory and then transfer it to the GPU, thus eliminating the overhead of double copying.

[0057] In one example, when the Load-to-GPU Pinning unit performs memory locking, it allocates a 10.83GB physical contiguous region in the host DRAM using the Linux system command `fallocate -l 10.83G / dev / lockmem`, mounts it as " / mnt / lockmem", and locks this region using `mlock( / mnt / lockmem, 10.83 * 1024^3)`, disabling paging by the operating system. It then copies the weights (32 layers, approximately 5GB total) of the Mixtral-8x7B MLP components from the " / model / mlp / " directory to " / mnt / lockmem / mlp / ", and the weights (32 layers, approximately 5.83GB total) of the MoE components from " / mod...". The file "el / moe / " is copied to " / mnt / lockmem / moe / ". During inference, zero-copy transfer is achieved using the PyTorch code `tensor=torch.load(' / mnt / lockmem / moe / moe_15 / expert_3.pt');tensor.to('cuda:0',non_blocking=True)`. The A100 GPU's DMA controller directly reads the weight data in " / mnt / lockmem". The transfer time is reduced from 15ms for synchronous copy to 3ms, eliminating the overhead of paging table lookup (about 2ms) and double copy (about 10ms). In this embodiment, Expert or Experts refers to expert components.

[0058] Optionally, the monitoring frequency of the computing resource monitoring unit and memory resource monitoring unit in the resource governance layer is real-time. The monitored data is used to obtain resource usage information, including CPU core utilization, peak GPU memory usage, and memory utilization of the edge devices.

[0059] In one possible implementation, all monitoring data is updated every 50ms (higher than the minimum requirement of "real-time") and fed back to the inference engine layer through the framework's resource scheduling interface: when the GPU memory usage rises to 19GB (not exceeding the 60% threshold of 24GB for the total 40GB of GPU memory), the inference engine layer maintains the current loading strategy; when the CPU usage reaches 75%, a low-priority logging process (non-inference related) is paused to reduce the CPU usage to below 70%, ensuring the supply of CPU resources for inference computing.

[0060] Example 3 On the other hand, an efficient reasoning method for large language models is provided for resource-constrained edge devices, applicable to the aforementioned LLM reasoning framework. The method includes: Step 101: Receive the terminal-side LLM inference request through the task scheduling layer of the framework, and decompose the inference task into sub-tasks adapted for dual-model collaborative processing by the inference task decomposition unit.

[0061] In this step, the edge device runs the E-SpecOffload framework. The inference request receiving unit of the task scheduling layer receives inference requests initiated by the user from the local client through the framework's API interface. The request type is "generating article summaries based on the Summeval dataset," and the specific input is the 35th article in the Summeval dataset (text length 800 characters), requiring the generation of a summary of no more than 200 characters.

[0062] Based on the logic of dual-model collaborative reasoning, and taking into account the length of the input text and the processing capacity of the model, the reasoning task decomposition unit decomposes the task into two sub-tasks.

[0063] Subtask A is the "draft model prediction subtask". The Mistral-7B draft model performs preliminary processing on the input article, generating a summary fragment of 8 tokens each time, which is represented as draft model maxnewtokens=8.

[0064] Subtask B is the "Target Model Validation Subtask," where the Mixtral-8x7B target model validates the correctness (including semantic coherence and information completeness) of the 8 token fragments generated by the draft model. If the validation passes, the next 8 token fragments are generated; if the validation fails, they are corrected and generated again. After decomposition, subtasks A and B are passed to the inference engine layer for execution in the order of "prediction-validation-re-prediction."

[0065] Step 102: The dual-batch interleaved pipeline unit of the inference engine layer controls the dual-model collaborative inference unit to perform draft model prediction and target model verification in a sequential interleaved manner. At the same time, the draft model unloading unit unloads the MLP components and KVcache of the draft model to the CPU memory, and the asynchronous loading unit prefetches the model components required for inference.

[0066] In one possible implementation, when the dual-model collaborative reasoning unit performs draft model prediction and target model verification in step 102, it adopts a double-batch interleaving process.

[0067] The double-batch interleaved processing involves, for two sequences to be inferred, the draft model first processes the prediction subtask of sequence 1, the target model simultaneously verifies the prediction result of sequence 2, and then alternately processes the prediction of sequence 2 and the verification of sequence 1.

[0068] For the dual-sequence inference task in steps 102 and 103, when loading the Mistral-7B and Mixtral-8x7B models, the inference engine layer directly uses the original bfloat16 precision weight file (without INT8 / INT4 quantization processing). The Transformer layer, MoE layer, and attention layer of the model all maintain the original structure (without channel pruning or layer distillation).

[0069] In this step, the inference engine layer first loads two models: the Mistral-7B draft model (bfloat16 precision, approximately 13GB in size) and the Mixtral-8x7B target model.

[0070] The dual-batch interleaved pipeline unit selects two inference sequences and controls the operation of the two models in an interleaved manner.

[0071] In one example, Sequence 1 is the Summeval article summary request from step 101, and Sequence 2 is a dialogue summary request from another Samsum dataset with a text length of 500 words.

[0072] In this example, the dual-model collaborative inference unit performs the dual-batch interleaving process as follows.

[0073] In the first time slice (0-50ms), the dual-model collaborative inference unit controls the draft model to process Sequence 1 and generate a summary fragment of the first 8 tokens ("This article focuses on the application of artificial intelligence in the medical field and analyzes the main scenarios for the current technology implementation"). At the same time, the draft model unloading unit unloads the MLP component of the draft model (approximately 2.8GB in size) and the KVcache of Sequence 1 (data volume O(16*800*4096), b=16) to the CPU memory. The asynchronous loading unit prefetches the MLP weights required for the next layer of inference of Sequence 1 (prefetched from CPU memory to GPU).

[0074] In the second time slice (50-100ms), the dual-model collaborative reasoning unit controls the target model to verify the 8-token fragment of sequence 1 (verification passed), while simultaneously controlling the draft model to process sequence 2 and generate a dialogue summary fragment of the first 8 tokens; the draft model unloading unit synchronously unloads the KVcache of sequence 2 to CPU memory; the asynchronous loading unit prefetches the MLP weights of the next layer of sequence 2.

[0075] The above interleaving logic is repeated in subsequent time slices until the digests of the two sequences are generated (Sequence 1 generates 192 tokens, and Sequence 2 generates 160 tokens).

[0076] In another example, we label two inference sequences as Sequence A and Sequence B, respectively, to illustrate the task. The inference task is to batch process dialogue summary requests from two Samsum datasets (Sequence A: "User A discusses order refund with customer service, 5-turn dialogue"; Sequence B: "User B plans weekend trip with friends, 8-turn dialogue"), with a hardware configuration of A100 GPU, Xeon CPU, and 1TB DRAM.

[0077] In the new example, the dual-model collaborative inference unit performs the dual-batch interleaving process as follows.

[0078] In the first phase (0-60ms), the draft model (Mistral-7B) processes sequence A, generating a summary fragment of 8 tokens based on the 5-turn dialogue text ("User A applied for a refund from customer service due to product quality issues"). Simultaneously, the target model (Mixtral-8x7B) processes the first 8 token fragments of sequence B (which were previously generated by the draft model), assigns features to 3 expert components through a gating network, and verifies whether the semantics of the fragments conform to the dialogue logic (verification passed). In the second phase (60-120ms), the draft model switches to sequence B and generates the next 8-token fragment based on the 8 rounds of dialogue text ("User B and his friends plan to go camping in the suburbs on Saturday"); at the same time, the target model switches to sequence A to verify the 8-token fragment generated by the draft model (confirming that "product quality problem" is consistent with the dialogue content, and the verification passes). In the third stage and thereafter, the above interleaved logic is repeated until sequence A generates a 120-token digest and sequence B generates a 180-token digest. Throughout the process, the two models have no idle waiting time, and computation and verification proceed in parallel.

[0079] After inference, the accuracy was validated using standard evaluation metrics (such as ROUGE-L and BLEU) on 10 sets of the Summeval dataset. The ROUGE-L value for the summary of Sequence 1 was 0.82, and the ROUGE-L value for the dialogue summary of Sequence 2 was 0.79, which is basically consistent with the original inference accuracy of the Mixtral-8x7B model on the cloud server (ROUGE-L values ​​of 0.83 and 0.80), with no loss of accuracy, thus verifying the effectiveness of "not adopting a model compression strategy".

[0080] Step 103: The resource governance layer monitors the CPU usage, GPU memory usage, and memory usage of the edge devices in real time, and feeds the monitoring data back to the inference engine layer to dynamically adjust the draft model unloading, asynchronous loading, and component loading strategies.

[0081] In this step, the resource governance layer monitors the hardware status in real time for the dual-sequence inference task in step 102, as detailed below.

[0082] Computational resource monitoring showed that CPU (Xeon Platinum 8378A) utilization rose to 68% when the draft model was unloaded and to 72% when the target model was validated; GPU (A100) memory usage was 17GB after loading the target model in the prefill stage, dropped to 9GB after unloading the target model in the decoding stage, and peaked at 19GB during the dual-sequence inference process after the draft model started working.

[0083] Memory resource monitoring showed that DRAM memory usage increased from 60GB to 140GB (out of a total of 1TB of memory) after unloading two sequences of KVcache.

[0084] The monitoring data is transmitted to the inference engine layer every 50ms through the framework's resource feedback channel. When the GPU memory usage reaches 19GB (not exceeding the 70% threshold of 28GB for the total 40GB of GPU memory), the inference engine layer maintains the current asynchronous loading speed. When the CPU utilization rises to 72% (close to the 75% warning threshold), the inference engine layer adjusts the draft model unloading strategy, delaying the next unloading action by 10ms until the CPU utilization drops to 68% before execution, thus avoiding CPU resource overload.

[0085] It should be noted that no model compression strategies such as quantization, pruning, and distillation are used during the inference process to maintain the original inference accuracy of LLM.

[0086] Optionally, a draft model unloading unit is used to asynchronously prefetch the weights of the current layer MLP components to GPU memory while the CPU performs attention calculations, thereby achieving overlap between calculations and weight transfers. In this case, attention calculations only transfer the current layer Query activation values. The data volume is O(b*s*d), where b is the batch size, s is the sequence length, and d is the hidden layer dimension.

[0087] The reason for transmitting only activation values ​​is as follows: In SpecOffload, the entire draft model resides in GPU memory, and its fixed memory usage accounts for a significant portion of the entire inference process, resulting in significantly lower GPU memory efficiency in edge environments compared to the existing baseline. Therefore, we are committed to completely removing the continuous GPU memory usage of the draft model to greatly reduce peak memory consumption.

[0088] E-SpecOffload offloads all MLP components and KVcache from the draft model to CPU memory, loading them layer by layer as needed for speculative execution. However, this operation introduces a new performance bottleneck: frequent KV cache migration. If attention computation is performed on the GPU, the entire KV cache (data size O(b*s*d), where b is the batch size, s is the sequence length, and d is the hidden layer dimension) needs to be migrated from CPU memory to GPU memory, resulting in high I / O latency that severely restricts decoding efficiency. Therefore, we dynamically delegate the attention computation task to the CPU, only needing to transfer the current layer activation value (Query, O(b*d)) from the GPU to the CPU, compute it, and then return the result to the GPU. This strategy reduces the KVcache I / O to 1 / s of the original (O(b*s*d) / O(b*d)=s). To further alleviate the MLP component loading latency, we introduce an inter-layer prefetching mechanism, asynchronously prefetching the current layer's MLP weights to GPU memory while computing attention, achieving computation-transfer overlap and further suppressing the latency caused by offloading. The asynchronous loading is almost simultaneous. Weights are loaded immediately once the CPU begins calculating the memory score.

[0089] In one example, when the draft model unloading unit performs unloading, it first migrates the Mistral-7B's MLP components (32 layers in total, with a total weight of approximately 2.8GB) and the KVcache of 16 batch tasks (total data volume O(16*512*4096) = 33,554,432 dimensions) to CPU memory. When entering the attention calculation stage, the CPU starts the attention score calculation (dot product operation between Query and KVcache), while the asynchronous loading unit prefetches the weights of the current layer (layer 10) of the MLP components (approximately 89MB) from the CPU memory to the GPU memory via the PCIe interface. Since the bandwidth of PCIeGen4x16 is 32GB / s, the weight prefetching takes approximately 2.8ms, while the CPU execution of attention calculation takes approximately 5ms, achieving complete overlap between calculation and transmission. The CPU finishes the attention calculation just as the prefetching is completed, and the GPU can immediately start the MLP layer calculation without any waiting delay.

[0090] Example 4 Optionally, when the inference engine layer performs fine-grained component loading, the method further includes: for each MoE layer of the target MoE architecture model, loading is performed in parallel by 9 independent threads, of which 1 thread loads the gated network component and 8 threads load 8 expert components respectively; after each component is loaded, the corresponding computation is started.

[0091] In one example, in this embodiment, the target model is Mixtral-8x7B (MoE architecture, each MoE layer contains 8 expert components), and the inference task is article summarization of the Summeval dataset (batch size 8).

[0092] When the inference engine layer performs fine-grained component loading, for the 20th MoE layer of Mixtral-8x7B, nine independent threads (thread IDs 1-9) are started. Thread 1 is bound to CPU core 3 and is responsible for loading the gated network component (weight file path is " / model / mixtral-8x7b / moe_20 / gate.pt", size approximately 0.5GB). Threads 2-9 are bound to CPU cores 4-11 respectively and are responsible for loading eight expert components (paths are " / model / mixtral-8x7b / moe_20 / expert_1.pt" to "expert_8.pt", each size approximately 1GB). (2GB); The gating network component of thread 1 is loaded at 20ms, immediately reads the input features of the current batch (8 samples, each with 4096 dimensions), and performs gating calculation (generating the weight allocation coefficients of 8 expert components); The expert component 1 of thread 2 is loaded at 35ms, immediately receives 20% of the feature data allocated by the gating network, and starts the expert component calculation (Feed-Forward network operation); The expert component 2 of thread 3 is loaded at 40ms, receives 15% of the feature data and starts the calculation; The expert components of subsequent threads 4-9 start the calculation in the order of loading completion (45ms, 50ms, 52ms, 55ms, 58ms, 60ms).

[0093] After all expert components have completed their calculations (at the latest 65ms), the output results of each expert component are aggregated and passed to the next layer for processing. The entire loading-calculation process has no overall waiting time, saving 30% of the time compared to synchronous loading.

[0094] In summary, the embodiments of this application also provide an LLM streaming method, which brings the following technical effects.

[0095] By employing techniques such as draft model unloading and Load-to-GPU Pinning, the throughput per unit of GPU memory usage (memory efficiency η=T / M) is the highest on the HumanEval, Summeval, and Samsum datasets, improving upon baselines such as SpecOffload and Fiddler by 1.02-2.25 times. It can adapt to the memory limitations of edge devices (e.g., an A100 GPU with 40GB VRAM can run the Mixtral-8x7B model smoothly).

[0096] Furthermore, since no compression strategies such as quantization, pruning, or distillation are used during the inference process, the original bfloat16 accuracy of the model is maintained. The ROUGE-L, BLEU, and other metrics of the inference results are basically consistent with the original model in the cloud, thus solving the accuracy loss problem caused by existing compression schemes. Technologies such as dual-batch interleaved pipelines, asynchronous loading, and fine-grained component loading enable inference throughput to meet the real-time requirements of the edge (e.g., the time to generate a digest of 256 tokens is controlled within 5 seconds). At the same time, resource monitoring and dynamic adjustment ensure that inference tasks do not affect other functions of the edge device (e.g., CPU utilization is kept below 75%, and memory usage does not exceed 20% of the total memory). Ultimately, this achieves an edge-side LLM inference experience that is "localized, low-latency, high-privacy, and high-precision," which aligns with the edge-side empowerment requirements of the OpenKylin intelligent engine.

[0097] Example 5 (Verification) This application also provides a test report on the E-SpecOffload inference framework.

[0098] To verify the advantages of E-SpecOffload in LLM edge deployment, we compared it with four state-of-the-art methods on three datasets. Experiments show that E-SpecOffload achieves the best memory efficiency. The specific experimental details are as follows.

[0099] In the experimental setup, the implementation involved building an E-SpecOffload inference framework based on SpecOffload on top of HuggingFaceTransformers v4.47.1.

[0100] We evaluated E-SpecOffload using the popular open-source model Mixtral-8x7B, which has 46.7B and a precision of bfloat16. The draft model used for speculative decoding is Mistral-7B. Although not evaluated, E-SpecOffload can support other models compatible with Transformers model classes.

[0101] The hardware requirements are as shown in Table 1, which is used to evaluate E-SpecOffload.

[0102] Table 1, Hardware Configuration

[0103] In the following code examples and illustrations, "component" will be translated as "Component" and "specification" as "Specification".

[0104] Datasets: The most common LLM benchmarks are used, which have different cue length distributions and tasks, as shown in Table 2.

[0105] Table 2, Dataset Configuration

[0106] In the following code examples and illustrations, dataset is translated as DataSet, task as Task, and size as Sizes.

[0107] Baselines: We compared our frameworks with four baseline frameworks, all designed to address GPU memory limitations: HuggingFaceAccelerate supports offloading weights from certain layers based on device mapping. We used version 1.5.2, hereinafter referred to as Accelerate. DeepSpeedZero-Inference supports offloading the entire model weights to the CPU or disk. We used version 0.16.1, hereinafter referred to as DeepSpeed. Fiddler strategically utilizes both CPU and GPU resources for MoE model inference. SpecOffload embeds SD into the offloading workflow, with draft models resident in GPU memory. All baseline frameworks natively support Mixtral.

[0108] Parameters: Batch size during inference is a critical parameter that affects throughput and latency. To evaluate the performance of edge devices under real-world operating conditions, we tested with batch sizes of 16, 32, and 64. This range was chosen to reflect common light, medium, and heavy load scenarios in edge applications, while also taking into account typical memory limitations on edge devices.

[0109] Key metric: Video memory efficiency, calculated as throughput divided by peak video memory usage (η=T / M).

[0110] The main test results are as follows.

[0111] Memory efficiency: Compared to the experimental baseline, our method demonstrates a significant advantage in memory efficiency. For example... Figure 1As shown, E-SpecOffload exhibits the highest memory efficiency across all datasets. Specifically, on the HumanEval dataset, E-SpecOffload achieves a memory efficiency of 1.02–1.83 times that of the baseline; on the SamSum dataset, it achieves 1.04–1.90 times the efficiency; and on the more challenging SummalEval dataset, its memory efficiency advantage is even more significant, reaching 2.25 times that of the baseline. These results demonstrate that E-SpecOffload can adapt to stricter memory constraints and achieve the highest throughput gain per unit of memory usage. This fully demonstrates its practical value in edge deployment scenarios.

[0112] Figures 4 to 6 The diagram shows a comparison of memory efficiency under different conditions. It can be seen that our inference does not employ model compression strategies such as quantization, pruning, and distillation, so there is virtually no loss in accuracy.

[0113] To clearly present the comparison results of different datasets, the datasets are vertically divided into three independent plots, labeled as follows: Figure 4 , Figure 5 , Figure 6 The horizontal and vertical axes of each subgraph have been translated into Chinese (the horizontal axis is "Generation Length (tokens)" and the vertical axis is "Memory Efficiency").

[0114] in, Figure 4 The graph shows the memory efficiency comparison of the code generation dataset (HumanEval), including the memory efficiency changes of the dataset under three load conditions of decoding batch sizes of 16, 32, and 64, corresponding to different generation lengths, clearly demonstrating the memory efficiency advantage of the method in this application in code generation tasks. Figure 5 The graph shows the comparison of memory efficiency for the text summarization dataset (Summeval), covering three conditions: decoding batch sizes of 16, 32, and 64. This allows for a direct observation of the memory efficiency performance of the proposed method in long text summarization tasks. Figure 6 The graph shows the comparison of memory efficiency for the dialogue summary dataset (Samsum), covering the three decoding batches mentioned above, demonstrating the versatility of the proposed method in dialogue-related tasks.

[0115] For experimental analysis, we used a progressive ablation strategy to analyze the contribution of each optimization point, as detailed in Table 3. Under the experimental configuration, the Mixtral8x7B model was deployed.

[0116] Table 3. Ablation experimental results on the HumanEval dataset.

[0117] Note: MO = DraftModelOffloading, LGP = Load-to-GPUPinning, FCL = Fine-grainedComponentLoading. Experimental configuration: prefillbatchsize = 8, decodingbatchsize = 32, draftbatchsize = 16, draftmaxnewtokens = 8, outputsizes = 256. The gray values ​​represent the percentage increase compared to the previous row. DraftModelOffloading translates to draft model offloading, Fine-grainedComponentLoading translates to fine-grained component loading, and Load-to-GPUPinning translates to GPU memory locking technology.

[0118] In the subsequent code examples and illustrations, memory efficiency is translated as Memoryefficiency and throughput is translated as Throughput.

[0119] We first evaluated the contribution of the "unload draft model" optimization separately. This mechanism works by unloading model parameters to CPU memory and dynamically prefetching them to the GPU on demand, combined with delegating attention computation to the CPU. Figure 2 As shown, enabling this feature alone significantly reduces the VRAM usage of the draft model (compare the curves of SpecOffload and E-SpecOffload in the graph). This directly results in an approximately 22.78% improvement in VRAM efficiency, proving the effectiveness of this optimization. More importantly, this optimization constitutes the main part of our overall VRAM reduction. Figure 3 As shown, after integrating all optimizations, the complete E-SpecOffload system ultimately achieved an overall reduction in video memory usage of over 50%. This result confirms the fundamental role of draft model offloading, which lays the foundation for subsequent optimizations (such as asynchronous loading) and enables the deployment of larger language models or long context processing on edge devices.

[0120] Figure 7A diagram comparing the dynamic memory usage of E-SpecOffload and SpecOffload during inference is shown. In the legend, S represents SpecOffload (speculation offload method), and E represents E-SpecOffload (the method of this application). These names are proprietary technical terms in the relevant technical fields, and some do not have a universally accepted Chinese translation; therefore, the English expressions are retained along with their corresponding Chinese definitions. The curve styles are differentiated as follows: the S series (SpecOffload) uses a dashed line style, covering S-Total (total memory usage of the speculation offload method), S-Target (memory usage of the target model of the speculation offload method), and S-Draft (memory usage of the draft model of the speculation offload method); the E series (E-SpecOffload) uses a solid line style, covering E-Total (total memory usage of the method of this application), E-Target (memory usage of the target model of the method of this application), and E-Draft (memory usage of the draft model of the method of this application). Figure 7 As can be seen from the curve distribution, the memory usage curves (S-Target and E-Target) of the two target models completely overlap, indicating that the method in this application does not adjust the inference logic of the target model and can maintain the same inference accuracy as the original model.

[0121] Figure 8 A diagram comparing memory usage across different frameworks is shown, with each curve in the diagram corresponding to a different inference framework. The purple curve represents SpecOffload, the red curve represents Fiddler, the blue curve represents DeepSpeed, the green curve represents Accelerate, and the cyan curve represents E-SpecOffload (the method described in this application). "Fiddler," "DeepSpeed," and "Accelerate" are proprietary names for inference frameworks already disclosed in the prior art, and there is no universally accepted Chinese translation; therefore, the English terms are retained to clarify the technical references. The pink area in the diagram represents the memory savings of E-SpecOffload compared to SpecOffload. Figure 8 As can be seen from the data, this application accelerates the GPU's access speed to CPU memory by locking the memory of frequently accessed components, reduces I / O latency in the CPU-GPU data transfer process, and increases throughput by 13.26% and memory efficiency by 11.86%. Finally, the introduction of fine-grained component parallel loading effectively reduces idle waiting in synchronous blocking loading, significantly reduces loading latency, and further increases throughput by 7.33% and memory efficiency by 7.37%.

[0122] In summary, the test results show that the optimization work in this application has a significant effect on improving video memory efficiency. Figure 7This verifies that the method in this application achieves effective optimization of video memory usage while maintaining inference accuracy without loss; Figure 8 This demonstrates E-SpecOffload's advantage in memory efficiency compared to existing frameworks. In LLM-oriented edge deployments, E-SpecOffload can maintain acceptable throughput while keeping memory usage low, showcasing its practical value in resource-constrained edge devices.

[0123] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0124] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above descriptions are merely optional embodiments of this application and are not intended to limit the application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A high-efficiency inference framework for large language model (LLM) for resource-constrained edge devices, characterized in that, The framework includes: The task scheduling layer is configured with an inference request receiving unit and an inference task decomposition unit, which are used to receive LLM inference requests from the end side and decompose the inference task into sub-tasks adapted for dual-model collaborative processing. The inference engine layer includes a dual-model collaborative inference unit, a dual-batch interleaved pipeline unit, a draft model unloading unit, and an asynchronous loading unit. The dual-model collaborative inference unit is used to perform draft model prediction and target model verification. The dual-batch interleaved pipeline unit is used to interleave two sequences to improve inference efficiency. The draft model unloading unit is used to unload the MLP components and KVcache of the draft model to CPU memory. The asynchronous loading unit is used to prefetch model components during the calculation process. The resource governance layer includes a computing resource monitoring unit and a memory resource monitoring unit, which are used to monitor the CPU utilization, GPU memory usage and memory usage of edge devices in real time, and communicate with the inference engine layer to dynamically adjust the allocation of inference resources.

2. The efficient inference framework for large language model LLM for resource-constrained edge devices according to claim 1, characterized in that, In the draft model unloading unit of the inference engine layer, the draft model is a Mistral type non-MoE architecture model. During the unloading process, the LLM attention calculation task is delegated to the CPU for execution, and the current layer activation value Query is transferred from the GPU to the CPU. After the calculation is completed, the result is returned to the GPU.

3. The efficient inference framework for large language model LLM for resource-constrained edge devices according to claim 1, characterized in that, The inference engine layer also includes a fine-grained component loading unit, which is used to prefetch gating networks and expert components in parallel using multi-threading technology for each MoE layer of the target MoE architecture model. The target MoE architecture model contains 8 expert components in each MoE layer. The multi-threading technology starts 9 threads to load 1 gating network and 8 expert components respectively. The expert components refer to independent model functional modules used to perform specific inference computing tasks.

4. The efficient inference framework for large language model LLM for resource-constrained edge devices according to claim 1, characterized in that, The inference engine layer also includes a Load-to-GPU Pinning unit, which is used to allocate a physically contiguous paged memory region in the host memory and persistently reside the weights of the MLP component and the MoE component in this region. The page lock memory area is 10.83GB in size and supports zero-copy pass-through through the tensor.to function for direct DMA access. The parameters of the tensor.to function include setting the device parameter and setting non_blocking=True. When the Load-to-GPUPinning unit performs memory locking, it includes: A 10.83GB physical contiguous region is allocated in the host memory as a page lock buffer. The weights of the MLP and MoE components are stored in this region. During inference, DMA achieves zero-copy transfer of component weights by directly accessing the page lock buffer, eliminating the overhead of paging table lookup and double copy.

5. The efficient inference framework for large language model LLM for resource-constrained edge devices according to claim 1, characterized in that, The monitoring frequency of the computing resource monitoring unit and memory resource monitoring unit in the resource governance layer is real-time; wherein, The monitored data is used to obtain resource usage information, including CPU core utilization, peak GPU memory usage, and memory usage of edge devices.

6. An efficient reasoning method for large language models for resource-constrained edge devices, applicable to the efficient reasoning framework for large language models for resource-constrained edge devices as described in any one of claims 1-5, characterized in that, The method includes: The task scheduling layer of the framework receives the LLM inference request from the end side, and the inference task decomposition unit decomposes the inference task into sub-tasks that are adapted for dual-model collaborative processing. The dual-batch interleaved pipeline unit of the inference engine layer controls the dual-model collaborative inference unit to perform draft model prediction and target model verification in a sequential interleaved manner. At the same time, the draft model unloading unit unloads the MLP components and KVcache of the draft model to the CPU memory, and the asynchronous loading unit prefetches the model components required for inference. The resource governance layer monitors the CPU usage, GPU memory usage, and memory usage of the edge devices in real time, and feeds the monitoring data back to the inference engine layer to dynamically adjust the draft model unloading, asynchronous loading, and component loading strategies.

7. The efficient reasoning method for large language models for resource-constrained edge devices according to claim 6, characterized in that, When the dual-model collaborative reasoning unit performs draft model prediction and target model verification, it adopts dual-batch interleaving processing. The double-batch interleaving process includes: For two sequences to be inferred, the draft model first processes the prediction subtask of sequence 1, while the target model simultaneously verifies the prediction result of sequence 2, and then alternately processes the prediction of sequence 2 and the verification of sequence 1.

8. The efficient reasoning method for large language models for resource-constrained edge devices according to claim 6, characterized in that, The draft model unloading unit is used to asynchronously prefetch the weights of the current layer MLP components to GPU memory while the CPU performs attention calculation, so as to achieve the overlap of calculation and weight transmission. The attention calculation only transmits the current layer Query activation value, and the data volume O conforms to the calculation rules about batch size b, sequence length s and hidden layer dimension d.

9. The efficient reasoning method for large language models for resource-constrained edge devices according to claim 6, characterized in that, When the inference engine layer performs fine-grained component loading, the method further includes: For each MoE layer of the target MoE architecture model, nine independent threads are used for parallel loading, with one thread loading the gated network component and eight threads loading eight expert components respectively. Each component starts its corresponding calculation after it is loaded.

Images