Hybrid expert large model inference system based on speculation and unloading

Abstract

A hybrid expert large model reasoning system based on speculation and unloading comprises a model deployer, an expert scheduler, a speculation controller and an execution engine, and is characterized in that when a quantized version of a target model is used as a draft model for speculation decoding, expert weight parameters are pre-fetched by utilizing an asynchronous prior expert activation mode; the generation speed of the MoE model is improved, and reasoning acceleration of the MoE model in different video memory resource scenes is realized.

Description

Technical Field

[0001] This invention relates to a technology in the field of large language models, specifically a large model reasoning system based on speculation and offloading of Mixture-of-Experts (MoE). Background Technology

[0002] While MoE models are computationally sparse, the large number of experts makes single-card deployment difficult. Existing offloading methods require prefetching experts but struggle with high-precision predictions without training; speculative decoding, lacking high-acceptance draft models, is also ill-suited for MoE. There is an urgent need for an inference acceleration solution that integrates training-free, high-precision expert prefetching with speculative decoding. Existing MOE deployment technologies cannot deploy hybrid expert models in scenarios with limited GPU memory, and they do not address system-level performance bottlenecks to achieve inference acceleration. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a hybrid expert large model inference system based on speculation and unloading. By using a quantized version of the target model as a draft model for speculative decoding, and simultaneously utilizing an asynchronous prior expert activation mode to prefetch expert weight parameters, the generation speed of the MoE model is improved, thereby accelerating the inference of the MoE model under different memory resource scenarios.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a hybrid expert large model inference system based on speculation and unloading, comprising: a model deployer, an expert scheduler, a speculation controller, and an execution engine, wherein: the model deployer realizes the quantization generation of draft models and the adjustment of parameter formats, as well as the deployment of draft models and target models in video memory and system memory; the expert scheduler dynamically manages the expert weight parameters in the hybrid expert model during its inference process according to the expert activation mode generated during the inference process of the draft model, so as to reduce or eliminate the waiting time for expert calculations to be performed during the verification stage of the target model; the speculation controller establishes a mathematical model of system resource parameters and speculative decoding algorithm parameters, and dynamically determines the optimal draft length; the execution engine reduces the kernel function startup overhead through kernel function fusion, quickly generates a draft sequence in the draft stage, and reorders expert calculations according to the expert activation mode in the verification stage, thereby realizing the parallel execution of the speculation workflow.

[0006] The quantization generation of the draft model refers to freezing the non-expert weights of the target model and quantizing only the expert weights of the target model, thereby sharing the non-expert parameters and further reducing system memory overhead.

[0007] The speculative decoding algorithm described above quickly generates multiple candidate tokens using a lightweight draft model, and then verifies these candidates in parallel by the target large model, accepting the correct results in batches, thereby reducing the number of autoregressive steps, improving throughput, and significantly accelerating the generation speed.

[0008] The optimal draft length refers to the maximum length k of the candidate sequences generated by the draft model in each round of speculative decoding, which is the upper bound of the number of acceptable tokens after parallel verification by the target model. If k is too large, the acceptance rate will decrease significantly, introducing redundant computation; if k is too small, the acceleration benefit will be limited. Therefore, there exists an optimal value of k that maximizes the overall inference acceleration efficiency.

[0009] The aforementioned dynamic determination of the optimal draft length refers to: performing a one-time performance profiling during the offline phase, experimentally measuring the impact of different draft lengths on latency, and determining the allowable draft length range that meets the service level objectives of first token latency and unit token generation speed; within this range, determining the optimal draft length value based on the ratio of expert weight prefetch time to kernel function calculation time.

[0010] The aforementioned expert activation pattern refers to the fact that during the forward inference process of a hybrid expert model, different token inputs activate different subsets of experts at each layer of the model. That is, the selection of experts varies across different tokens in a single inference task, and also across different layers of the same token. This activation pattern represents prior knowledge and can optimize the scheduling of expert weight parameters during the inference process. Specifically, the expert activation pattern of the draft model is highly similar to that of the target model; therefore, the pattern of the draft model can be used as the prediction result to pre-fetch the expert weight parameters of the target model.

[0011] The aforementioned asynchronous parallel expert weight parameter prefetching refers to the transfer of expert weight parameters between memory and video memory during inference, which is achieved through asynchronous flow control. This operation can be executed in parallel with the computation process of MoE inference. Technical effect

[0012] This invention effectively alleviates the memory bottleneck and I / O latency issues caused by the large parameter scale of the MoE model through a collaborative design of speculative execution and expert offloading mechanisms. It utilizes a lightweight quantized draft model to predict future expert activation patterns, driving high-precision asynchronous prefetching and hiding PCIe data transfer overhead within the computation process. This invention eliminates the need for retraining, completing model adaptation offline and significantly lowering the deployment threshold. Experiments on mainstream models such as Phi-MoE show that compared to naive offloading schemes, this invention achieves up to 5.9 times inference speedup; compared to fine-grained offloading frameworks, it still improves throughput by 1.8 times. In cloud service scenarios, system throughput is improved by 19.9%; on edge devices, end-to-end latency is reduced by 10.36%. This invention not only enables the MoE model to run efficiently on general-purpose hardware but also constructs a novel decoupled architecture design paradigm oriented towards data-dependent access. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the invention;

[0014] Figure 2 This is a schematic diagram of the system of the present invention;

[0015] Figure 3 This is a flowchart of the offline phase of an example.

[0016] Figure 4 Here is a flowchart of the online inference stage for an example;

[0017] Figure 5 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0018] like Figure 1 and Figure 2 As shown in the figure, this embodiment relates to a hybrid expert large model inference system based on speculation and unloading, including: a model deployer, an expert scheduler, a speculation controller, and an execution engine.

[0019] The model deployer includes a model quantization module, a parameter packaging module, and an expert unloading module. The model quantization module generates a speculatively decodable quantized draft model from the target model. The parameter packaging module packages the modified weight parameters of the draft model into a fusion expert format suitable for calculation by the kernel function fusion module in the execution engine. The expert unloading module initializes the draft model and the target model into the hardware environment. Due to GPU memory (VRAM) limitations, some experts of the target model need to be unloaded to CPU memory (RAM).

[0020] The quantization draft model refers to the MoE model obtained by retaining the non-expert part and quantizing only the expert part, so as to minimize the GPU memory usage while maintaining the model quality by sharing the non-expert weight parameters with the target model.

[0021] The aforementioned fusion expert format refers to: concatenating the discrete expert module weight parameters into a high-dimensional matrix according to a new dimension for kernel function fusion calculation.

[0022] The expert scheduler includes a prefetch module and a loading module. The prefetch module loads the expert parameters that may be needed in the future from the CPU memory to the video memory in advance, based on the draft length k provided by the inference controller and the expert activation mode of the draft model, thus hiding the blocking and latency caused by loading the expert weight parameters. When the prediction fails, the loading module selects the optimal expert parameters for eviction based on the performance impact assessment and loads the experts that are actually needed, ensuring that experts on the critical path are available.

[0023] The inference controller includes a performance profiling module, a service target module, and a parameter solving module. The performance profiling module performs offline performance profiling before real task inference to obtain fine-grained system evaluation indicators for the draft model and the target model to optimize system configuration. The service target module determines the maximum allowable draft length k that satisfies the target TTFT (Time-To-First-token) budget. The parameter solving module solves for the optimal k that maximizes throughput efficiency in real time, taking into account the current hardware load.

[0024] The fine-grained system evaluation metrics include, but are not limited to, expert module computation latency, loading latency, and model token reception rate during model inference.

[0025] The execution engine includes a kernel function fusion module, a computation flow optimization module, and a speculative decoder module. The kernel function fusion module applies optimization strategies such as operator fusion, mixed-precision computation, device flow synchronization, and context isolation at the underlying execution level to improve the execution efficiency of the draft model. The computation flow optimization module dynamically adjusts the execution order of expert calculations during the verification process based on the expert activation mode and the distribution of expert hardware to maintain the stable performance of the target model verification. The speculative decoder module implements the speculative decoding algorithm to complete the end-to-end generation of the task.

[0026] like Figure 3 and Figure 4 As shown, this embodiment uses a hybrid expert large model inference method based on the above system, including:

[0027] Step 1, Offline Stage Figure 3 As shown, it specifically includes:

[0028] 1.1 Use the model deployer to perform expert part quantization on the target model to obtain a complete draft model to be processed;

[0029] 1.2 The expert weight parameters of the draft model are calculated using the kernel function format to obtain the draft model;

[0030] 1.3 Deploy both the draft model and the target model to GPU memory and CPU memory;

[0031] 1.4 Perform a one-time inference test to analyze performance and obtain system performance metrics;

[0032] 1.5 Obtain user SLO service targets such as first-word meta-delay (TTFT) and first-word meta-delay (TPOT).

[0033] Step 2, Online Phase Figure 4 As shown, it specifically includes:

[0034] 2.1 Accepts external input task prompts and generates control commands;

[0035] 2.2 Generate a single batch task processing flow based on the speculative decoding paradigm;

[0036] The single-batch task processing flow refers to the generation process following the speculative decoding algorithm, that is, first using a draft model to generate candidates and then using a target model to verify them, repeating multiple rounds of iteration to generate the final output sequence.

[0037] 2.3 Generate the optimal draft length k based on the system's operating status;

[0038] 2.4 The draft model generates a sequence of candidate tokens based on the prefix input and the optimal draft length k;

[0039] 2.5 During the generation process, the expert weight parameters of the target model are prefetched and loaded under the guidance of the expert scheduler;

[0040] 2.6 The target model verifies the candidate sequence and obtains the received sequence, backoff prefix, and synchronizes the model state;

[0041] 2.7 Repeat steps 2.3-2.6 until the task is completed.

[0042] Through practical application experiments, a test environment was built on a server equipped with an Intel® Xeon® Silver 4310 CPU @ 2.10GHz and an NVIDIA® A100-PCIE-40GB GPU. The specific software packages used were Python 3.11, PyTorch 2.7.0, and CUDA 12.6. The hybrid expert large model inference system based on speculation and unloading of this invention was run with different test configurations (dataset, model, and memory size limitations). The generation speed was tested on a language model evaluation dataset, and the results are as follows: Figure 3As shown in the figure. Experiments verified the effectiveness of the proposed method for improving the inference speed of the MoE model (up to 2.34 times speed improvement, with improvements across different experimental configurations), demonstrating that the system can maintain sufficient inference acceleration under different models and datasets.

[0043] In the experiments described above, generating fixed-length sequences given a prefix length was the task of the inference system. The experiments used GSM8K, wikitext, human-eval, GPQA, and C4 as evaluation datasets, covering different task difficulties and professional domains, with tests covering over 500 test cases. Three different hybrid expert models were tested: Qwen2-MoE-A2.7B, deepseek-v2-lite, and Phi-MoE.

[0044] This embodiment simulates the system's performance by building an end-to-end inference service platform, using two MoE offloading baselines—mixtral-offloading and transformers—as controls. Figure 5 The four technical solutions are represented as follows: (a) the hybrid expert large model inference system based on speculation and offloading of this invention; (b) mixtral-offloading-SM, a version of the baseline method for high-speed inference with equivalent memory constraints achieved through expert-level fine-grained offloading; (c) mixtral-offloading-SC, an implementation of the baseline method for high-speed inference with equivalent cache size achieved through expert-level fine-grained offloading; and (d) transformers, a mainstream open-source inference framework, uses a larger-granularity hierarchical parameter offloading scheme. The results are as follows: Figure 5 As shown in the figure. Through specific experimental verification, the technology involved in this invention has achieved good experimental results. This experiment uses the time per output token (TPOT) as an indicator, and it can be seen that the experimental effect of this invention is significantly improved compared to other methods.

[0045] Compared to the current state-of-the-art technology, Mixtral-Offloading-SM, this invention achieves a speedup of up to 2.34 times in inference speed, and up to 9.8 times compared to mainstream commercial frameworks like Transformers. The inference framework not only demonstrates significant speedup across different methods but also maintains good consistency and average speedup across multiple datasets. Even with limited GPU memory, this embodiment still exhibits good speedup performance.

[0046] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A hybrid expert large model inference system based on speculation and offloading, characterized in that, The application relates to a model deployment device, an expert scheduler, a speculation controller and an execution engine, wherein: the model deployment device realizes the quantitative generation of a draft model, the adjustment of a parameter format and the deployment of the draft model and a target model in a display memory and a memory; the expert scheduler dynamically manages the expert weight parameters in a mixed expert model during the inference process of the draft model according to an expert activation mode generated in the inference process, so as to reduce or eliminate the waiting time of the expert calculation in the target model verification stage; the speculation controller establishes a mathematical model of system resource parameters and a speculative decoding algorithm parameter, and dynamically determines an optimal draft length; and the execution engine reduces the kernel function startup overhead through kernel function fusion, quickly generates a draft sequence in the draft stage, reorders the expert calculation according to the expert activation mode in the verification stage, and realizes parallel execution of the speculation workflow. The quantitative generation of the draft model refers to freezing the non-expert weights of the target model and only quantizing the expert weights of the target model, so as to share the non-expert parameters and further reduce the system display memory overhead. The optimal draft length refers to the maximum length k of the candidate sequence generated by the draft model in each round of speculative decoding, that is, the upper limit of the acceptable token quantity after parallel verification of the target model. If k is too large, the acceptance rate will significantly decrease, and redundant calculation will be introduced. If k is too small, the acceleration benefit is limited. Therefore, there is an optimal k value that maximizes the overall inference acceleration efficiency.

2. The hybrid expert large model inference system based on speculation and offload according to claim 1, wherein, The dynamic determination of the optimal draft length refers to performing one-time performance profiling in the offline stage, measuring the influence of different draft lengths on the delay through experiments, and determining the allowed draft length interval that meets the first token delay and unit token generation speed service level targets according to the influence. In the interval, the optimal value of the draft length is determined according to the expert weight pre-fetching time and the proportion of kernel function calculation time. The expert activation mode refers to the fact that different token inputs activate different expert subsets at each layer of the mixed expert model during the forward inference process, that is, the selection of experts differs between different tokens in a single inference task and between different layers of the same token. This activation mode is prior knowledge that optimizes the scheduling of expert weight parameters in the inference process. The expert activation mode of the draft model is greatly similar to that of the target model, so the expert weight parameters of the target model are pre-fetched as the prediction result of the draft model. The asynchronous parallel expert weight parameter pre-fetching refers to the transmission of the expert weight parameters between the memory and the display memory through asynchronous flow control during the inference process. The operation is performed in parallel with the calculation process of the MoE inference.

3. The hybrid expert large model inference system based on speculation and offload according to claim 1, wherein, ​ ​ 4. The hybrid speculation and offload based expert large model inference system of any of claims 1-3, wherein, The model deployer comprises a model quantization module, a parameter packing module and an expert offloading module, wherein: the model quantization module generates a speculatively decodable quantized draft model from a target model; the parameter packing module packs the weight parameter modification format of the draft model into a fusion expert format suitable for calculation of a kernel function fusion module in an execution engine; and the expert offloading module initializes the draft model and the target model to a hardware environment, and due to GPU video memory (VRAM) limitation, needs to offload part of the experts of the target model to CPU memory (RAM).

5. The hybrid expert large model inference system based on speculation and offload according to claim 4, wherein, The quantized draft model refers to an MoE model obtained by retaining a non-expert part and quantizing only an expert part, so as to reduce the video memory occupation as much as possible by sharing the non-expert weight parameters with the target model while maintaining the model quality. The fusion expert format refers to splicing discrete expert module weight parameters into a high-dimensional matrix according to a new dimension for kernel function fusion calculation.

6. The hybrid speculation and offload based expert large model inference system of any one of claims 1-3, wherein, The expert scheduler comprises a prefetch module and a loading module, wherein: the prefetch module loads the expert parameters that may be needed in the future from the CPU memory to the video memory in advance according to the draft length k and the expert activation mode of the draft model provided by the speculation controller, so as to hide the blocking and delay caused by the expert weight parameter loading; and the loading module selects the optimal expert parameters to be evicted based on performance impact evaluation when prediction fails, loads the actual required experts, and ensures that the experts on the critical path are available.

7. The hybrid speculation and offload based expert large model inference system of any one of claims 1-3, wherein, The speculation controller comprises a performance profiling module, a service target module and a parameter solving module, wherein: the performance profiling module performs offline performance profiling before real task inference, obtains fine-grained system evaluation index optimization system configuration of the draft model and the target model, determines the maximum allowed draft length k that satisfies the target TTFT (Time-To-First-token) budget, and solves the optimal k that maximizes the throughput efficiency in real time in combination with the current hardware load.

8. The hybrid speculation and offload based expert large model inference system of any one of claims 1-3, wherein, The execution engine comprises a kernel function fusion module, a computation flow optimization module and a speculation decoder module, wherein: the kernel function fusion module applies optimization strategies such as operator fusion, mixed precision calculation, device stream synchronization and context isolation at the bottom execution level, so as to improve the execution efficiency of the draft model; the computation flow optimization module dynamically adjusts the expert calculation execution order in the verification process according to the expert activation mode and the expert hardware distribution situation, so as to maintain the stable performance of the target model verification; and the speculation decoder module implements the speculation decoding algorithm and completes the end-to-end generation of the task.

9. A hybrid expert model inference method based on the system of any one of claims 1-8, characterized in that, The method comprises: Step 1, an offline stage, specifically comprising: 1.1 quantizing the expert part of the target model by using the model deployer to obtain a complete draft model to be processed; 1.2 organizing the expert weight parameters of the draft model into a calculation kernel function format to obtain the draft model; 1.3 deploying the draft model and the target model into the GPU video memory and the CPU memory; 1.4 running one-time inference to perform performance analysis and obtain system performance indexes; 1.5 obtaining user SLO service targets such as first token delay (TTFT) and per token delay (TPOT); Step 2, an online stage, specifically comprising: 2.1 accept external input task prompt word (prompt) and generate control instruction; 2.2 generate single batch task processing flow according to speculative decoding paradigm, that is, follow the generation process of speculative decoding algorithm, that is, first generate candidate with draft model and then verify with target model, repeat multiple rounds of iteration to generate final output sequence; 2.3 generate optimal draft length k based on system running state; 2.4 draft model generates candidate token sequence based on prefix input and optimal draft length k; 2.5 pre-fetch and load expert weight parameters of target model under the guidance of expert dispatcher during generation process; 2.6 target model verifies the candidate sequence and gets the received sequence, backtracks the prefix, and synchronizes the model state; 2.7 repeat 2.3-2.6 until the task processing is completed.

Images