Memory-limited device MoE large model inference optimization system and method based on double prediction

Abstract

The application discloses a memory-restricted device MoE large model inference optimization system and method based on double prediction, and belongs to the technical field of MoE large model inference optimization. In order to solve the problem of optimizing the inference performance of a MoE-based LLM on a memory-restricted device, the application comprises a layer-by-layer predictor, a token-by-token predictor, and an expert management module, wherein the expert management module comprises an expert cache, a temporary expert buffer and an I / O scheduler. The layer-by-layer predictor is connected to the token-by-token predictor, the layer-by-layer predictor and the token-by-token predictor are respectively connected to the I / O scheduler, the token-by-token predictor is connected to the expert cache, and the I / O scheduler is respectively connected to the temporary expert buffer and the expert cache. The application innovatively proposes a predictive expert cache, a temporary expert buffer and a token-by-token prefetching technology; and the application enables the inference system to use less memory resources and significantly improves the inference speed of the LLM.

Description

Technical Field

[0001] This invention belongs to the field of MoE large model inference optimization technology, specifically relating to a MoE large model inference optimization system and method based on dual prediction for memory-constrained devices. Background Technology

[0002] MoE-based LLMs effectively alleviate the deployment challenges of traditional LLMs on memory-constrained devices through their unique sparse architecture design. Their core advantage lies in their dynamic parameter activation mechanism: the model divides the overall parameters into multiple expert modules, activating only a small number of relevant experts for each input sample, while other expert parameters remain dormant. This not only reduces the computational load required for the inference model, but more importantly, this "on-demand loading" characteristic reduces memory usage to at most 10%-20% of that of a full-parameter model. This makes it possible for large models to run efficiently on memory-constrained terminal devices such as IoT devices and smartphones, opening a new path for the widespread deployment of generative AI technology. Since early 2025, several MoE-based LLMs, such as Deepseek-R1, Grok 3, and Qwen3-MoE, have been launched both domestically and internationally. These models have received considerable attention and application, successfully demonstrating the advantages of the MoE architecture.

[0003] For memory-constrained devices, the dynamic parameter activation mechanism of the MoE large model can be used to offload parameters. Most model parameters only need to be stored in the model file on solid-state storage, thus avoiding loading all model parameters (up to 10GB) into the limited memory. Figure 1 As shown, the MoE large model has multiple Transformer layers. The first half of each layer is MHA (Multi-Head Attention), and the second half is FFN (Feedforward Neural Network). During inference, FFN requires multiplying the input vector with a large parameter matrix. The large parameter matrix used by FFN accounts for the vast majority of all parameters in the entire model. The MoE architecture divides the large parameter matrix of FFN into N experts. Before computing FFN, a gating function selects K experts from all experts based on the output vector of MHA. When the parameters of these K experts are loaded into memory on demand, the vector is input into each of the K experts for computation. The results are then weighted and summed to obtain the output of FFN. This reduces the computational load and parameter usage compared to the non-MoE architecture to approximately [missing information]. To reduce the significant impact of I / O during expert parameter loading on inference performance, inference systems with parameter unloading capabilities generally need to adopt the following two basic strategies:

[0004] Expert prefetching. To parallelize computation and I / O, when computing the preceding Transformer layers, methods such as MLP can be used to predict the experts used in subsequent layers and load them into memory in advance.

[0005] Expert caching. To reduce I / O data volume, when inferring multiple tokens in LLM, since the use of experts in the same Transformer layer has a certain temporal locality, experts used by previous tokens can be cached in memory and managed using replacement algorithms such as LRU.

[0006] Song et al. proposed ProMoE, which uses a two-layer MLP to predict the experts used in layer i+2. The input of this MLP is the output vector of MHA, and the output is the selection probability of N experts. Experts ranked in the top K and not in the layer i+2 cache are prefetched. For expert caching, ProMoE uses a conventional LRU management approach. When a cache miss occurs, a non-blocking miss handling method is adopted: the inference calculation of experts already in the cache is executed first, and I / O is performed in parallel on the experts whose caches are missing. Furthermore, ProMoE introduces two priorities for I / O operations: cache miss I / O has higher priority than prefetch I / O. By transmitting prefetch I / O in blocks, it can be preempted by cache miss I / O.

[0007] Zhong et al. proposed HybriMoE, which implements an expert cache replacement algorithm. It scores each expert in the cache and evicts the expert with the lowest score during cache replacement. The expert score is an exponentially weighted average of the gating function outputs from each previous layer of reasoning for each token. This reflects the importance of different experts during the reasoning process, essentially similar to frequency-based cache replacement, but incorporating the specific reasoning process of LLM.

[0008] Currently, MoE parameter offloading technologies such as ProMoE generally adopt a layer-by-layer expert prediction / prefetching approach. These technologies consider the inter-layer correlations when inferring a token using LLM, which does indeed achieve a high prediction accuracy (80-90%). However, layer-by-layer prediction does not utilize the correlations between different tokens (belonging to similar language contexts) in the inference process, and layer-by-layer prefetching also has significant limitations: if the distance between the prediction layer and the target layer is too large, the prediction accuracy is low; if the distance is too small, there is insufficient time for prefetching; if all experts predicted at the target layer are in the cache, I / O will be idle, and overly advanced prefetching cannot be performed. Furthermore, ProMoE and HybriMoE use historical inference information to manage expert caches, employing simple strategies to determine whether an expert should be evicted from the cache. These strategies are actually unreasonable, and their hit rates differ significantly from the theoretically optimal cache management strategy (evicting the most recently used expert), failing to fully utilize the device's limited memory. Summary of the Invention

[0009] The problem this invention aims to solve is to optimize the inference performance of MoE-based LLM on memory-constrained devices and improve the efficiency of device memory utilization. It proposes an optimization system and method for MoE large model inference on memory-constrained devices based on dual prediction.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A memory-constrained device MoE large model inference optimization system based on dual prediction includes a layer-by-layer predictor, a token-by-token predictor, and an expert management module. The expert management module includes an expert cache, a temporary expert buffer, and an I / O scheduler.

[0012] The layer-by-layer predictor is connected to the token-by-token predictor, the layer-by-layer predictor and the token-by-token predictor are respectively connected to the I / O scheduler, the token-by-token predictor is connected to the expert cache, and the I / O scheduler is respectively connected to the temporary expert buffer and the expert cache;

[0013] The layer-by-layer predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) to predict the probability of the next layer expert being selected. The prediction is then fed into the I / O scheduler to prefetch the expert to be used in the next layer prediction.

[0014] A token-by-token predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) used to calculate the relative scores of experts and input them into the I / O scheduler for expert priority comparison.

[0015] An expert cache is set up in each Transformer layer to cache the experts used in each Transformer layer. The expert cache performs expert replacement based on the output of the token-by-token predictor.

[0016] A temporary expert buffer, shared by all Transformer layers, is used to cache a portion of the experts currently being inferred from the layer and a portion of the experts that will be used by the next layer.

[0017] The I / O scheduler accepts the outputs of the layer-by-layer predictor and the token-by-token predictor, then retrieves the highest-priority I / O request from the I / O queue and executes it, reading the expert parameters of the execution result into the expert cache or temporary expert buffer.

[0018] Furthermore, the input layer of the 2-layer Multilayer Perceptron (MLP) is set to d. model Dimension, hidden layer is d mid The output layer is N-dimensional, the activation function between two layers is ReLU, and the total number of parameters is d. model ×d mid +d mid ×N, where dmodel And N are the model parameters of the target LLM itself, d mid Set the value to 100~300.

[0019] Furthermore, the expert cache in the expert management module is used to cache the experts used at each level. The expert cache is set as a two-dimensional array Cache[i][j], which represents the j-th expert item in the i-th level expert cache. Cache[i] is the i-th level expert cache. Each level expert cache has c expert items, where c is the expert cache capacity parameter, set to 10%~20% of N based on the target device's memory capacity. Each expert item is a triple (v, e, a, s), where v is the valid bit, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded.

[0020] The temporary expert buffer in the expert management module is shared by all layers. It only temporarily stores experts that need to be used immediately but are not suitable for storage in the expert cache. The temporary expert buffer is essentially an array Ubuf[] with 2K expert items. Each expert item is a quintuple (v, l, e, a, s), where v is the valid bit, l is the layer number, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded.

[0021] In the expert management module, each item in the I / O scheduler queue of the I / O scheduler is a six-tuple (p, t, a′, o, s, S), representing an I / O request. Here, p is the priority, t is the first enqueue time, a′ is the target memory buffer address of the I / O, o is the external storage location, s is the number of bytes already loaded, and S is the total number of bytes on external storage.

[0022] A method for inference optimization of a memory-constrained device MoE large model based on dual prediction, implemented using the aforementioned system for inference optimization of a memory-constrained device MoE large model based on dual prediction, includes the following steps:

[0023] S1. Offline pre-training stage: Deploy the layer-by-layer predictor and the token-by-token predictor on the MoE large model, and train the layer-by-layer predictor and the token-by-token predictor for each Transformer layer;

[0024] S2. Online Inference Phase: Based on the layer-by-layer predictor and token-by-token predictor trained in step S1, the expert management module is embedded into the LLM inference system. The expert management module completes the online inference phase through five steps: initialization, prediction execution, hit check, expert acquisition, and unit completion, and obtains the selected expert parameters. Before the LLM inference system executes each expert inference, the expert management module passes the selected expert parameters loaded into memory to the LLM inference system.

[0025] Furthermore, the specific implementation method of step S1 includes the following steps:

[0026] S1.1. Select a dataset containing multiple questions, input each question from the dataset into the MoE large model, causing it to generate a total of m tokens, and capture the multi-head attention output vector x of the i-th Transformer layer during inference of the MoE large model. i The set of expert IDs actually selected by the i-th Transformer layer in the MoE large model during inference, E i For each layer, the vector sequence {x} is obtained. ’ i,k} and set sequence {E ’ i,k}, where x ’ i,k E is the MHA output vector of the i-th layer when inferring the k-th token in LLM. ’ i,k For the set of expert IDs actually selected at the i-th layer when inferring the k-th token in LLM;

[0027] S1.2. Set the predictor f of the i-th layer i The training set for (x) is {(x)} ’ i,k , y i,k )}, where y i,k Let y be the data feature of the i-th layer predictor. i,k The definition of is:

[0028]

[0029] in, For the data features of the i-th layer predictor of the j-th expert, Let this be the set of expert IDs actually selected at the (i+1) mod n level when inferring the k-th token in LLM. The total number of experts at each level of the LLM;

[0030] S1.3. Set the i-th layer token predictor g i The training set for (x) is {(x)} ’i,k , z i,k )}, where z i,k For the data features of the i-th layer token predictor, in calculating z i,k First, calculate the next usage time u for each expert. i,k,j The expression is:

[0031]

[0032] Then, press all experts j Sort them from smallest to largest to get their relative rankings v i,k,j Based on this, z is calculated i,k The expression is:

[0033]

[0034] in, Let be the data features of the i-th layer token predictor of the j-th expert, and let α be the expert selection factor, set to a real number between 1 and 3;

[0035] S1.4. For each layer's layer-by-layer predictor and token-by-token predictor, construct a training set of size m, train 2n MLPs, use the loss function BCEWithLogitsLoss, set a random deactivation of 0.01 in the hidden layer, and train until the model accuracy converges.

[0036] Furthermore, the specific method for initialization in step S2 is as follows:

[0037] Load the MLP parameters of all layers' layer-by-layer predictors and token-by-token predictors into memory;

[0038] Initialize the expert cache and temporary expert buffer. For all Cache[i][j].a and Ubuf[i].a, allocate a buffer accessible to the LLM inference system computation backend, where Cache[i][j].a is the memory buffer address of the cached expert parameter of the j-th expert item in the i-th level expert cache, and Ubuf[i].a is the memory buffer address of the cached expert parameter of the i-th expert item in the temporary expert buffer.

[0039] Based on the LLM model file metadata, obtain the storage location and size information of all experts, and initialize Expert[i][j];

[0040] Initialize Score[i][j] to all zeros;

[0041] Initialize the execution threads for the I / O scheduler, layer-by-layer predictor, and token-by-token predictor.

[0042] Furthermore, the specific method for performing the prediction in step S2 is as follows:

[0043] After initialization, the LLM inference system begins inference layer by layer on multiple tokens. After the LLM inference system completes the inference of the i-th layer of MHA, it obtains the output vector x. i This data is then passed to the expert management module, which then performs the prediction, following the process shown below:

[0044] Perform token-by-token prediction on the CPU side to calculate g. i (x i ), update the vector to the scoreScore[i] of the expert with the number j in the i-th layer;

[0045] Perform layer-by-layer prediction on the CPU side to calculate f. i (x i Press f i (x i [e] Enumerate the K expert IDs e in descending order;

[0046] If e is not in the Cache[(i+1) mod n], then perform a layer-by-layer prefetch;

[0047] If Score[(i+1) mod n][e] < If Score[(i+1) mod n][j]}, then e should not be loaded into the expert cache, and an expert item should be allocated for it in the temporary expert buffer; otherwise, a replacement should be performed in the cache[(i+1) mod n], and an expert item should be allocated for it.

[0048] The layer-by-layer prefetch request for e (2, t, a, Expert[(i+1) mod n][e].offset, 0, Expert[(i+1) mod n][e].size) is placed into the I / O scheduler queue IOQue; if an I / O request for e already exists in IOQue, the priority of the layer-by-layer prefetch request for e is increased to 2 and then placed back into IOQue.

[0049] Furthermore, the specific method for the hit check in step S2 is as follows:

[0050] After performing the prediction, the LLM inference system performs the calculation of the i-th level expert selection, obtaining the set of expert numbers E for the i-th level selection. i It is then passed to the expert management module, which, according to E i To perform a hit check, follow the procedure shown below:

[0051] Enumerate all queue items in IOQue corresponding to experts at level i. If the expert corresponding to it is not in level E... i If the condition is met, the queue item will be removed from the IOQueue; otherwise, its priority will be increased to 1.

[0052] If the currently executing I / O request is not in E i If the i-layer expert is notified, the I / O scheduler will remove it.

[0053] Enumerate e∈E in descending order of Score[i][e]. i If e is neither in Cache[i] nor in the temporary expert buffer, then layer-by-layer prefetching is required; if Score[i][e] < If Score[i][j]}, then e should not be loaded into the expert cache, and an expert item should be allocated for it in the temporary expert buffer; otherwise, a replacement should be performed in Cache[i], and an expert item should be allocated for it.

[0054] Place the I / O request (1, t, a, Expert[i][e].offset, 0, Expert[i][e].size) obtained during the hit check process into the IOQue.

[0055] Furthermore, the specific method for obtaining expert information in step S2 is as follows:

[0056] After the expert management module checks for a hit, the LLM inference system begins executing K expert inferences at layer i. For each expert inference, it needs to retrieve a loaded expert from the expert management module. At this time, the expert management module performs the following process:

[0057] Enumerate the expert entries Cache[i][j] in the i-th level expert cache. If e = Cache[i][j].e ∈ E i If expert e has not been computed and Cache[i][j].s = Expert[i][e].size, then expert e has been fully loaded, and the expert parameter address Cache[i][j].a is passed to the LLM inference system;

[0058] If expert e is not found, enumerate the expert item Ubuf[j] in the temporary expert buffer; if Ubuf[j].l=i and e=Ubuf[j].e∈E i If expert e has not been computed and Ubuf[j].s = Expert[i][e].size, then expert e has been fully loaded, and the expert parameter address Ubuf[j].a is passed to the LLM inference system.

[0059] If no loaded but uncomputed expert is found above, block and wait for the completion of the highest-priority layer-i I / O request in the IOQue, and then pass the request address to the LLM inference system.

[0060] Further, the specific method completed by the unit in step S2 is as follows:

[0061] When the LLM inference system successfully obtains an expert and finishes inferring all K experts selected for layer i, it notifies the expert management module that the single-layer inference is completed. At this time, the expert management module executes the following process:

[0062] Enumerate the expert items Ubuf[j] in the temporary expert buffer. If Ubuf[j].l = i, then set Ubuf[j].v = 0, that is, invalidate the layer-i experts in the temporary expert buffer.

[0063] Enumerate the top min(c, αK) experts e with the largest Score[i][e]. Let their ranking in Score[i] be r (0 ≤ r < N). If e is not in Cache[i], then perform expert cache replacement, allocate an expert item with address a, and put the per-token prefetch request (r + 3, t, a, Expert[i][e].offset, 0, Expert[i][e].size) into the IOQue, where t is the current time.

[0064] Advantages of the present invention:

[0065] The MoE large model inference optimization system for memory-constrained devices based on dual prediction described in the present invention introduces a per-token predictor, allowing prefetching of experts for subsequent tokens in the expert cache for each layer in advance; adopts a prediction-based expert cache management algorithm, which has higher memory utilization efficiency and can achieve a hit rate close to the theoretical optimum; adopts the design of a temporary expert buffer, allowing experts to be used without residing in the cache, further improving the cache efficiency; uses multi-priority preemptive I / O processing, dividing different prefetch operations into multiple fixed priorities, and enhancing the I / O efficiency.

[0066] This invention presents an optimization method for MoE large-scale inference on memory-constrained devices based on dual prediction. Building upon existing expert caching and expert prefetching techniques, it innovatively proposes predictive expert caching, temporary expert buffers, and per-token prefetching. The predictive expert caching and temporary expert buffers allow the expert cache hit rate to approach its theoretical upper limit, improving the utilization efficiency of limited memory resources. Per-token prefetching allows for more advanced expert prefetching, ensuring that I / O bandwidth is not idle, further enhancing the parallelism of computation and I / O. Experiments based on the latest MoE large-scale models demonstrate that these key techniques enable the inference system to use fewer memory resources, significantly improving the inference speed of LLM. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the structure of a memory-constrained device MoE large model inference optimization system based on dual prediction, as described in this invention.

[0068] Figure 2 The flowchart is a process for optimizing MoE large model inference for memory-constrained devices based on dual prediction, as described in this invention.

[0069] Figure 3 This is a structural diagram of the predictor MLP of the present invention;

[0070] Figure 4 This is a schematic diagram illustrating the interaction between the expert management module and the LLM inference system.

[0071] Figure 5 This is the LLM inference flowchart of the present invention;

[0072] Figure 6 This is a graph comparing the inference speed of the models. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0074] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0075] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 - Appendix Figure 6 Detailed explanation is as follows:

[0076] Example 1:

[0077] A memory-constrained device MoE large model inference optimization system based on dual prediction includes a layer-by-layer predictor, a token-by-token predictor, and an expert management module. The expert management module includes an expert cache, a temporary expert buffer, and an I / O scheduler.

[0078] The layer-by-layer predictor is connected to the token-by-token predictor, the layer-by-layer predictor and the token-by-token predictor are respectively connected to the I / O scheduler, the token-by-token predictor is connected to the expert cache, and the I / O scheduler is respectively connected to the temporary expert buffer and the expert cache;

[0079] The layer-by-layer predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) to predict the probability of the next layer expert being selected. The prediction is then fed into the I / O scheduler to prefetch the expert to be used in the next layer prediction.

[0080] A token-by-token predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) used to calculate the relative scores of experts and input them into the I / O scheduler for expert priority comparison.

[0081] An expert cache is set up in each Transformer layer to cache the experts used in each Transformer layer. The expert cache performs expert replacement based on the output of the token-by-token predictor.

[0082] A temporary expert buffer, shared by all Transformer layers, is used to cache a portion of the experts currently being inferred from the layer and a portion of the experts that will be used by the next layer.

[0083] The I / O scheduler accepts the outputs of the layer-by-layer predictor and the token-by-token predictor, then retrieves the highest-priority I / O request from the I / O queue and executes it, reading the expert parameters of the execution result into the expert cache or temporary expert buffer.

[0084] Furthermore, the input layer of the 2-layer Multilayer Perceptron (MLP) is set to d. model Dimension, hidden layer is d mid The output layer is N-dimensional, the activation function between two layers is ReLU, and the total number of parameters is d. model ×d mid +d mid ×N, where d model And N are the model parameters of the target LLM itself, d mid Set the value to 100~300.

[0085] Furthermore, the expert cache in the expert management module is used to cache the experts used at each level. The expert cache is set as a two-dimensional array Cache[i][j], which represents the j-th expert item in the i-th level expert cache. Cache[i] is the i-th level expert cache. Each level expert cache has c expert items, where c is the expert cache capacity parameter, set to 10%~20% of N based on the target device's memory capacity. Each expert item is a triple (v, e, a, s), where v is the valid bit, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded.

[0086] The temporary expert buffer in the expert management module is shared by all layers. It only temporarily stores experts that need to be used immediately but are not suitable for storage in the expert cache. The temporary expert buffer is essentially an array Ubuf[] with 2K expert items. Each expert item is a quintuple (v, l, e, a, s), where v is the valid bit, l is the layer number, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded.

[0087] In the expert management module, each item in the I / O scheduler queue of the I / O scheduler is a six-tuple (p, t, a′, o, s, S), representing an I / O request. Here, p is the priority, t is the first enqueue time, a′ is the target memory buffer address of the I / O, o is the external storage location, s is the number of bytes already loaded, and S is the total number of bytes on external storage.

[0088] Example 2:

[0089] A method for optimizing MoE (MoE) large model inference for memory-constrained devices based on dual prediction, implemented using the dual prediction-based MoE large model inference optimization system described in Example 1, includes the following steps:

[0090] S1. Offline pre-training stage: Deploy the layer-by-layer predictor and the token-by-token predictor on the MoE large model, and train the layer-by-layer predictor and the token-by-token predictor for each Transformer layer;

[0091] Furthermore, the specific implementation method of step S1 includes the following steps:

[0092] S1.1. Select a dataset containing multiple questions, and input each question from the dataset into the MoE large model to generate a total of m tokens, where m is set to 500000. Capture the multi-head attention output vector x of the i-th Transformer layer during inference of the MoE large model. i The set of expert IDs actually selected by the i-th Transformer layer in the MoE large model during inference, E i For each layer, the vector sequence {x} is obtained. ’ i,k} and set sequence {E ’ i,k}, where x ’ i,k E is the MHA output vector of the i-th layer when inferring the k-th token in LLM. ’ i,k For the set of expert IDs actually selected at the i-th layer when inferring the k-th token in LLM;

[0093] S1.2. Set the predictor f of the i-th layer i The training set for (x) is {(x)} ’ i,k , y i,k )}, where y i,k Let y be the data feature of the i-th layer predictor. i,k The definition of is:

[0094]

[0095] in, For the data features of the i-th layer predictor of the j-th expert, Let this be the set of expert IDs actually selected at the (i+1) mod n level when inferring the k-th token in LLM. The total number of experts at each level of the LLM;

[0096] S1.3. Set the i-th layer token predictor g i The training set for (x) is {(x)} ’ i,k , z i,k )}, where z i,k For the data features of the i-th layer token predictor, in calculating z i,k First, calculate the next usage time u for each expert. i,k,j The expression is:

[0097]

[0098] Then, press all experts j Sort them from smallest to largest to get their relative rankings v i,k,j Based on this, z is calculated i,k The expression is:

[0099]

[0100] in, Let be the data features of the i-th layer token predictor of the j-th expert, and let α be the expert selection factor, set to a real number between 1 and 3;

[0101] Furthermore, a smaller α increases the accuracy of per-token prefetching but decreases the hit rate of the expert cache. When α=1, the per-token predictor essentially predicts the expert to be used for the next token.

[0102] S1.4. For each layer's layer-by-layer predictor and token-by-token predictor, construct a training set of size m, train 2n MLPs, use the loss function BCEWithLogitsLoss, set a random deactivation of 0.01 in the hidden layer, and train until the model accuracy converges.

[0103] S2. Online Inference Phase: Based on the layer-by-layer predictor and token-by-token predictor trained in step S1, the expert management module is embedded into the LLM inference system. The expert management module completes the online inference phase through five steps: initialization, prediction execution, hit check, expert acquisition, and unit completion, and obtains the selected expert parameters. Before the LLM inference system executes each expert inference, the expert management module passes the selected expert parameters loaded into memory to the LLM inference system.

[0104] For the online inference phase, this embodiment will implement the expert management module by embedding it into the existing LLM inference system (such as llama.cpp). The interaction between the expert management module and the LLM inference system is as follows: Figure 4 As shown, the LLM inference system needs to process the obtained x when inferring each Transformer layer. i and E i The parameters are passed to the expert management module, and the expert management module is notified when the i-level inference is completed. The expert management module then passes the expert parameters loaded into memory to the LLM inference system when needed.

[0105] The LLM inference process after implementing the expert management module in this embodiment is as follows: Figure 5As shown, the green part represents what the expert management module needs to implement, while the white part is inherent to the LLM inference system and is irrelevant to this invention. First, the LLM inference system and the expert management module are initialized, and then inference begins from the first token. For each token, inference is performed sequentially from layer 0 to n-1. When inferring the i-th Transformer layer, the MHA is inferred first to obtain the output vector x. i Then x i The results are passed to the expert management module for prediction, then the i-th layer expert selection (i.e., the gating function of MoE) is inferred, and the calculated expert selection set E is used. i The data is passed to the expert management module. Next, inference begins with K experts. Each time, the system first retrieves a loaded expert parameter from the expert management module, then performs the expert inference. When all K experts have completed their inference, the system notifies the expert management module. This process continues until all tokens have been inferred. The expert management module in this embodiment requires specific implementation. Figure 5 The five processes listed below depend on three components: the expert cache, the temporary expert buffer, and the I / O scheduler. The implementation of these three components and the five processes is described in detail below.

[0106] Furthermore, the expert cache has the following property: when a new expert needs to be loaded into the i-level expert cache, an empty expert entry needs to be allocated for it. This requires the following process:

[0107] If there exists j such that Cache[i][j].v=0, then allocate Cache[i][j].

[0108] Otherwise, allocate Cache[i][ The cache cache will remove the expert with the smallest score[i][j]. Specifically, if E is known at this point... i If layer i is not fully computed, then Cache[i][j].e∈E is not considered during eviction. i Experts.

[0109] Furthermore, the temporary expert buffer has the following property: for all expert items that satisfy Ubuf[j].v=1, Ubuf[j].l always has only two values: i and (i+1) mod n (where i is a certain layer). Therefore, a temporary expert buffer with a capacity of 2K is sufficient to accommodate all experts used by two adjacent layers, without needing to consider cache replacement.

[0110] Furthermore, the I / O scheduler executes asynchronously throughout the entire LLM inference process, and its execution process is as follows:

[0111] Step 1. Wait for IOQue to be non-empty;

[0112] Step 2. Take out the queue item (p, t, a, o, s, S) with the earliest order in IOQue;

[0113] Step 3. Calculate the data volume size of a single I / O. If p = 1, this I / O request cannot be preempted, and let size = S - s; if p = 2, let size = min(B, S - s); otherwise, let size = min(B / 8, S - s). Here, the parameter B is the I / O transfer block size (in bytes), at least 4KB. A larger B can achieve better I / O transfer efficiency but reduces the preemptibility of low-priority I / O operations;

[0114] Step 4. Execute the I / O operation, read size bytes from the external storage o + s to the memory address a + s, and block waiting for the I / O operation to complete;

[0115] Step 5. Let s = s + size. If s ≥ S, then this expert is completely loaded into the memory, and go to Step 1;

[0116] Step 6. If this expert is not completely loaded, then perform a check: If the I / O scheduler is notified to cancel the currently ongoing I / O operation, discard this queue item and go to Step 1; if IOQue is non-empty and <p, that is, there is an I / O request with a higher priority in the queue, then perform preemption, put this queue item back into IOQue, and go to Step 2;

[0117] Step 7. If the current I / O request is not discarded or preempted, go to Step 3 and continue the unfinished I / O for it.

[0118] Furthermore, the specific method for initialization in Step S2 is:

[0119] Load the MLP parameters of the layer-by-layer predictors and token-by-token predictors of all layers into the memory;

[0120] Initialize the expert Cache and the temporary expert buffer. For all Cache[i][j].a and Ubuf[i].a, allocate buffers accessible by the LLM inference system computing backend, where Cache[i][j].a is the memory buffer address of the cached expert parameters of the j-th expert item in the i-th layer of the expert Cache, and Ubuf[i].a is the memory buffer address of the cached expert parameters of the i-th expert item in the temporary expert buffer;

[0121] According to the metadata of the LLM model file, obtain the storage location and size information of all experts, and initialize Expert[i][j];

[0122] Initialize Score[i][j] to all zeros;

[0123] Initialize the execution threads for the I / O scheduler, layer-by-layer predictor, and token-by-token predictor.

[0124] Furthermore, the specific method for performing the prediction in step S2 is as follows:

[0125] After initialization, the LLM inference system begins inference layer by layer on multiple tokens. After the LLM inference system completes the inference of the i-th layer of MHA, it obtains the output vector x. i This data is then passed to the expert management module, which then performs the prediction, following the process shown below:

[0126] Perform token-by-token prediction on the CPU side to calculate g. i (x i ), update the vector to the scoreScore[i] of the expert with the number j in the i-th layer;

[0127] Perform layer-by-layer prediction on the CPU side to calculate f. i (x i Press f i (x i [e] Enumerate the K expert IDs e in descending order;

[0128] If e is not in the Cache[(i+1) mod n], then perform a layer-by-layer prefetch;

[0129] If Score[(i+1) mod n][e] < If Score[(i+1) mod n][j]}, then e should not be loaded into the expert cache, and an expert item should be allocated for it in the temporary expert buffer; otherwise, a replacement should be performed in the cache[(i+1) mod n], and an expert item should be allocated for it.

[0130] The layer-by-layer prefetch request for e (2, t, a, Expert[(i+1) mod n][e].offset, 0, Expert[(i+1) mod n][e].size) is placed into the I / O scheduler queue IOQue; if an I / O request for e already exists in IOQue, the priority of the layer-by-layer prefetch request for e is increased to 2 and then placed back into IOQue.

[0131] Furthermore, the specific method for the hit check in step S2 is as follows:

[0132] After performing the prediction, the LLM inference system performs the calculation of the i-th level expert selection, obtaining the set of expert numbers E for the i-th level selection. i It is then passed to the expert management module, which, according to E i To perform a hit check, follow the procedure shown below:

[0133] Enumerate all queue items in IOQue corresponding to experts at level i. If the expert corresponding to it is not in level E... i If the condition is met, the queue item will be removed from the IOQueue; otherwise, its priority will be increased to 1.

[0134] If the currently executing I / O request is not in E i If the i-layer expert is notified, the I / O scheduler will remove it.

[0135] Enumerate e∈E in descending order of Score[i][e]. i If e is neither in Cache[i] nor in the temporary expert buffer, then layer-by-layer prefetching is required; if Score[i][e] < If Score[i][j]}, then e should not be loaded into the expert cache, and an expert item should be allocated for it in the temporary expert buffer; otherwise, a replacement should be performed in Cache[i], and an expert item should be allocated for it.

[0136] Place the I / O request (1, t, a, Expert[i][e].offset, 0, Expert[i][e].size) obtained during the hit check process into the IOQue.

[0137] Furthermore, the specific method for obtaining expert information in step S2 is as follows:

[0138] After the expert management module checks for a hit, the LLM inference system begins executing K expert inferences at layer i. For each expert inference, it needs to retrieve a loaded expert from the expert management module. At this time, the expert management module performs the following process:

[0139] Enumerate the expert entries Cache[i][j] in the i-th level expert cache. If e = Cache[i][j].e ∈ E i If expert e has not been computed and Cache[i][j].s = Expert[i][e].size, then expert e has been fully loaded, and the expert parameter address Cache[i][j].a is passed to the LLM inference system;

[0140] If expert e is not found, enumerate the expert item Ubuf[j] in the temporary expert buffer; if Ubuf[j].l=i and e=Ubuf[j].e∈Ei If expert e has not been calculated and Ubuf[j].s = Expert[i][e].size, it means that expert e has been fully loaded. Then, the address of the expert parameter Ubuf[j].a is passed to the LLM inference system.

[0141] If no uncalculated expert that has been loaded is found as above, block and wait for the highest-priority i-layer I / O request in IOQue to complete, and then pass the address of this request to the LLM inference system.

[0142] Furthermore, the specific method completed by the unit in step S2 is as follows:

[0143] When the LLM inference system successfully obtains the experts and finishes inferring all K experts selected in the i-th layer, it notifies the expert management module that the single-layer inference is completed. At this time, the expert management module performs the following process:

[0144] Enumerate the expert items Ubuf[j] in the temporary expert buffer. If Ubuf[j].l = i, then set Ubuf[j].v = 0, that is, invalidate the experts in the i-th layer of the temporary expert buffer.

[0145] Enumerate the top min(c, αK) experts e with the largest Score[i][e]. Suppose their ranking in Score[i] is r (0 ≤ r < N). If e is not in Cache[i], then perform expert Cache replacement, allocate an expert item with address a, and put the per-token prefetch request (r + 3, t, a, Expert[i][e].offset, 0, Expert[i][e].size) into IOQue, where t is the current time.

[0146] Apply this embodiment to the llama.cpp inference system and conduct experiments using the currently latest released MoE large model Qwen3-30B-A3B (Q8 quantization, size 30GB). The parameter settings are shown in Table 1:

[0147] Table 1 Parameter Settings

[0148]

[0149] When training the predictor, use Chinese-DeepSeek-R1-Distill-data-110k [3]A randomly selected question from the Chinese dataset is input into the LLM, outputting 600,000 tokens, which are used to construct the training set. During inference, the expert cache capacity c is set to 5~24 (memory usage approximately 2~7GB). A randomly selected question from this dataset (not in the training set) is used as input to the LLM, and the inference speed during the Decode phase (outputting tokens) is measured. The experiment uses a Jetson Orin NX development board (with 16GB LPDDR5 unified memory and a PCIe 4.0 x4 NVMe SSD, using a CUDA compute backend) as the hardware.

[0150] The experiment was divided into five groups:

[0151] LC: LRU expert cache only, no expert prefetching.

[0152] PC: Only predictive expert cache (and temporary expert buffer), no expert prefetching.

[0153] S+LC: Layer-by-layer prefetching + LRU expert cache, where layer-by-layer prefetching is the prediction of layer i by (i+2) mod n layers.

[0154] This is the implementation of ProMoE, which differs from the present invention.

[0155] L+PC: Layer-by-layer prefetching + predictive expert cache (and temporary expert buffer), this is the invention.

[0156] Some of the technologies in this paper do not involve per-token prefetching.

[0157] L+T+PC: Layer-by-layer prefetching + token-by-token prefetching + predictive expert cache (and temporary expert buffer)

[0158] (area), which is the complete technology in this embodiment.

[0159] Figure 6 The inference speeds of each experimental group under different expert cache capacity settings are shown. It can be found that L+T+PC (in this embodiment) has a significantly higher inference speed than S+LC (ProMoE). When c=5, the inference speed of L+T+PC is 57% higher than that of S+LC. Comparing L+T+PC and L+PC, the gain of per-token prefetching on inference speed can be observed. When c=12, the inference speed of L+T+PC is 12% higher than that of L+PC. Comparing LC and PC, it can be found that predictive cache has better performance than LRU in the absence of expert prefetching.

[0160] Key term definitions:

[0161] 1. LLM (Large Language Model): A deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text, and to a certain extent simulates the human language cognition and generation process.

[0162] 2. MoE (Mixture of Expert): A deep learning model architecture that divides the deep learning model into different experts, with the model input being computed by some of these experts, in order to reduce the amount of computation and parameters used.

[0163] 3. Transformer: An attention-based deep learning model architecture widely used in large language models, which are typically composed of multiple stacked Transformer layers.

[0164] 4. FFN (Feedforward Neural Network): A fully connected basic neural network structure that accounts for most of the parameters in large language models.

[0165] 5. MHA (Multi-Head Attention): Multi-head attention is an extension of the attention mechanism widely used in Transformer models. It obtains the attention distribution of different subspaces of the input sequence by running multiple independent attention mechanisms in parallel, thereby capturing more comprehensive potential semantic associations in the sequence.

[0166] 6. MLP (Multilayer Perceptron): A multilayer perceptron is a basic neural network structure with an input layer, hidden layers, and an output layer, with each layer fully connected to the previous one.

[0167] 7. A token refers to a basic word unit in natural language. The reasoning of a large language model is based on tokens. The input of the entire model is the previous token of the natural language text, and the output after reasoning is the next token.

[0168] 8. LRU (Least Recently Used): A commonly used cache replacement algorithm that evicts the least recently used element from the cache each time.

[0169] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A memory-constrained device MoE large model inference optimization system based on dual prediction, characterized in that, It includes a layer-by-layer predictor, a token-by-token predictor, and an expert management module, wherein the expert management module includes an expert cache, a temporary expert buffer, and an I / O scheduler; The layer-by-layer predictor is connected to the token-by-token predictor, the layer-by-layer predictor and the token-by-token predictor are respectively connected to the I / O scheduler, the token-by-token predictor is connected to the expert cache, and the I / O scheduler is respectively connected to the temporary expert buffer and the expert cache; The layer-by-layer predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) to predict the probability of the next layer expert being selected. The prediction is then fed into the I / O scheduler to prefetch the expert to be used in the next layer prediction. A token-by-token predictor is set in each Transformer layer, which is a 2-layer multilayer perceptron (MLP) used to calculate the relative scores of experts and input them into the I / O scheduler for expert priority comparison. An expert cache is set up in each Transformer layer to cache the experts used in each Transformer layer. The expert cache performs expert replacement based on the output of the token-by-token predictor. A temporary expert buffer, shared by all Transformer layers, is used to cache a portion of the experts currently being inferred from the layer and a portion of the experts that will be used by the next layer. The I / O scheduler accepts the outputs of the layer-by-layer predictor and the token-by-token predictor, then retrieves the highest-priority I / O request from the I / O queue and executes it, reading the expert parameters of the execution result into the expert cache or temporary expert buffer.

2. The MoE large model inference optimization system for memory-constrained devices based on dual prediction as described in claim 1, characterized in that, The input layer of a two-layer multilayer perceptron (MLP) is set to d. model Dimension, hidden layer is d mid The output layer is N-dimensional, the activation function between two layers is ReLU, and the total number of parameters is d. model ×d mid +d mid ×N, where d model And N are the model parameters of the target LLM itself, d mid Set the value to 100~300.

3. A memory-constrained device MoE large model inference optimization system based on dual prediction as described in claim 1 or 2, characterized in that, The expert cache in the expert management module is used to cache the experts used at each level. The expert cache is set as a two-dimensional array Cache[i][j], which represents the j-th expert item in the i-th level expert cache. Cache[i] is the expert cache of the i-th level. Each level of expert cache has c expert items, where c is the expert cache capacity parameter, which is set to 10%~20% of N based on the target device's memory capacity. Each expert item is a triple (v, e, a, s), where v is the valid bit, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded. The temporary expert buffer in the expert management module is shared by all layers. It only temporarily stores experts that need to be used immediately but are not suitable for storage in the expert cache. The temporary expert buffer is essentially an array Ubuf[] with 2K expert items. Each expert item is a quintuple (v, l, e, a, s), where v is the valid bit, l is the layer number, e is the expert number, a is the memory buffer address for caching expert parameters, and s is the number of bytes already loaded. In the expert management module, each item in the I / O scheduler queue of the I / O scheduler is a six-tuple (p, t, a′, o, s, S), representing an I / O request. Here, p is the priority, t is the first enqueue time, a′ is the target memory buffer address of the I / O, o is the external storage location, s is the number of bytes already loaded, and S is the total number of bytes on external storage.

4. A method for inference optimization of a memory-constrained device MoE large model based on dual prediction, implemented using the memory-constrained device MoE large model inference optimization system based on dual prediction as described in any one of claims 1-3, characterized in that, Includes the following steps: S1. Offline pre-training stage: Deploy the layer-by-layer predictor and the token-by-token predictor on the MoE large model, and train the layer-by-layer predictor and the token-by-token predictor for each Transformer layer; S2. Online Inference Phase: Based on the layer-by-layer predictor and token-by-token predictor trained in step S1, the expert management module is embedded into the LLM inference system. The expert management module completes the online inference phase through five steps: initialization, prediction execution, hit check, expert acquisition, and unit completion, and obtains the selected expert parameters. Before the LLM inference system executes each expert inference, the expert management module passes the selected expert parameters loaded into memory to the LLM inference system.

5. The method for optimizing MoE large model inference for memory-constrained devices based on dual prediction as described in claim 4, wherein the specific implementation method of step S1 includes the following steps: S1.

1. Select a dataset containing multiple questions, input each question from the dataset into the MoE large model, causing it to generate a total of m tokens, and capture the multi-head attention output vector x of the i-th Transformer layer during inference of the MoE large model. i The set of expert IDs actually selected by the i-th Transformer layer in the MoE large model during inference, E i For each layer, the vector sequence {x} is obtained. ’ i,k } and set sequence {E ’ i,k },in, x ’ i,k E is the MHA output vector of the i-th layer when inferring the k-th token in LLM. ’ i,k For the set of expert IDs actually selected at the i-th layer when inferring the k-th token in LLM; S1.

2. Set the predictor f of the i-th layer. i The training set for (x) is {(x)} ’ i,k , y i,k )}, where y i,k Let y be the data feature of the i-th layer predictor. i,k The definition of is: ； in, For the data features of the i-th layer predictor of the j-th expert, Let this be the set of expert IDs actually selected at the (i+1) mod n level when inferring the k-th token in LLM. The total number of experts at each level of the LLM; S1.

3. Set the i-th layer token predictor g i The training set for (x) is {(x)} ’ i,k , z i,k )}, where z i,k For the data features of the i-th layer token predictor, in calculating z i,k First, calculate the next usage time u for each expert. i,k,j The expression is: ； Then, press all experts j Sort them from smallest to largest to get their relative rankings v i,k,j Based on this, z is calculated i,k The expression is: ； in, Let be the data features of the i-th layer token predictor of the j-th expert, and let α be the expert selection factor, set to a real number between 1 and 3; S1.

4. For each layer's layer-by-layer predictor and token-by-token predictor, construct a training set of size m, train 2n MLPs, use the loss function BCEWithLogitsLoss, set a random deactivation of 0.01 in the hidden layer, and train until the model accuracy converges.

6. The method for optimizing MoE large model inference for memory-constrained devices based on dual prediction as described in claim 5, wherein the specific method for initialization in step S2 is as follows: Load the MLP parameters of all layers' layer-by-layer predictors and token-by-token predictors into memory; Initialize the expert cache and temporary expert buffer. For all Cache[i][j].a and Ubuf[i].a, allocate a buffer accessible to the LLM inference system computation backend, where Cache[i][j].a is the memory buffer address of the cached expert parameter of the j-th expert item in the i-th level expert cache, and Ubuf[i].a is the memory buffer address of the cached expert parameter of the i-th expert item in the temporary expert buffer. Based on the LLM model file metadata, obtain the storage location and size information of all experts, and initialize Expert[i][j]; Initialize Score[i][j] to all zeros; Initialize the execution threads for the I / O scheduler, layer-by-layer predictor, and token-by-token predictor.

7. The method for optimizing MoE large model inference for memory-constrained devices based on dual prediction as described in claim 6, wherein the specific method for performing prediction in step S2 is as follows: After initialization, the LLM inference system begins inference layer by layer on multiple tokens. After the LLM inference system completes the inference of the i-th layer of MHA, it obtains the output vector x. i This data is then passed to the expert management module, which then performs the prediction. The execution process is as follows: Perform token-by-token prediction on the CPU side to calculate g. i (x i ), update the vector to the scoreScore[i] of the expert with the number j in the i-th layer; Perform layer-by-layer prediction on the CPU side to calculate f. i (x i Press f i (x i [e] Enumerate the K expert IDs e in descending order; If e is not in the Cache[(i+1) mod n], then perform a layer-by-layer prefetch; If Score[(i+1) mod n][e] < If Score[(i+1) mod n][j]}, then e should not be loaded into the expert cache, and an expert item should be allocated for it in the temporary expert buffer; otherwise, a replacement should be performed in Cache[(i+1) mod n], and an expert item should be allocated for it. The layer-by-layer prefetch request for e (2, t, a, Expert[(i+1) mod n][e].offset, 0, Expert[(i+1) mod n][e].size) is placed into the I / O scheduler queue IOQue; if an I / O request for e already exists in IOQue, the priority of the layer-by-layer prefetch request for e is increased to 2 and then placed back into IOQue.

8. For an optimized method for MoE large model inference on memory - constrained devices based on dual prediction according to claim 7, the specific method for hit check in step S2 is as follows: After performing the prediction, the LLM inference system performs the calculation of the i-th level expert selection, obtaining the set of expert numbers E for the i-th level selection. i It is then passed to the expert management module, which, according to E i The hit check is performed as follows: Enumerate all queue items in IOQue corresponding to experts at level i. If the expert corresponding to it is not in level E... i If the condition is met, the queue item will be removed from the IOQueue; otherwise, its priority will be increased to 1. If the currently executing I / O request is not in E i If the i-layer expert is notified, the I / O scheduler will remove it. Enumerate e∈E in descending order of Score[i][e]. i If e is neither in Cache[i] nor in the temporary expert buffer, then layer-by-layer prefetching is required; if Score[i][e] < If Score[i][j]}, then e should not be loaded into the expert cache; instead, an expert item should be allocated for it in the temporary expert buffer. Otherwise, perform replacement in Cache[i] and allocate an expert item for it; Put the I / O request (1, t, a, Expert[i][e].offset, 0, Expert[i][e].size) for e obtained in the hit check process into IOQue.

9. For an optimized method for MoE large model inference on memory - constrained devices based on dual prediction according to claim 8, the specific method for expert acquisition in step S2 is as follows: After the expert management module checks for a hit, the LLM inference system starts to perform K expert inferences for layer i. For each expert inference, it needs to obtain a loaded expert from the expert management module. At this time, the process of the expert management module is as follows: Enumerate the expert entries Cache[i][j] in the i-th level expert cache. If e = Cache[i][j].e ∈ E i If expert e has not been computed and Cache[i][j].s = Expert[i][e].size, then expert e has been fully loaded, and the expert parameter address Cache[i][j].a is passed to the LLM inference system; If expert e is not found, enumerate the expert item Ubuf[j] in the temporary expert buffer; if Ubuf[j].l=i and e=Ubuf[j].e∈E i If expert e has not been computed and Ubuf[j].s = Expert[i][e].size, then expert e has been fully loaded, and the expert parameter address Ubuf[j].a is passed to the LLM inference system; If no un - calculated expert that has been loaded is found, block and wait for the highest - priority layer - i I / O request in IOQue to complete, and then pass the request address to the LLM inference system.

10. For an optimized method for MoE large model inference on memory - constrained devices based on dual prediction according to claim 9, the specific method for unit completion in step S2 is as follows: When the LLM inference system successfully obtains an expert and finishes inferring all K experts selected for layer i, it notifies the expert management module that single - layer inference is completed. At this time, the process of the expert management module is as follows: Enumerate the expert items Ubuf[j] in the temporary expert buffer. If Ubuf[j].l = i, then set Ubuf[j].v = 0, that is, invalidate the layer - i experts in the temporary expert buffer; Enumerate the top min(c, αK) experts e with the largest Score[i][e]. Let their rank in Score[i] be r (0 ≤ r < N). If e is not in Cache[i], then perform expert cache replacement, allocate an expert item with address a, and put the per - token pre - fetch request (r + 3, t, a, Expert[i][e].offset, 0, Expert[i][e].size) into IOQue, where t is the current time.

Images