A Hybrid Expert Large Language Model with Expert Replacement: End-Side Scheduling and Inference Method and System
By employing expert replacement and load balancing scheduling strategies on edge devices, the problems of inference latency and accuracy under limited video memory were solved, achieving efficient utilization of computing resources and preservation of model accuracy, and improving the response speed and accuracy of edge AI devices.
Patent Information
- Application Number
- CN202611326730.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-30
- Publication Date
- 2026-11-17
AI Technical Summary
On memory-constrained edge devices, existing technologies struggle to maintain high accuracy of hybrid expert models while reducing inference latency, and existing scheduling strategies fail to effectively utilize the differences in contributions from different experts, resulting in wasted transmission bandwidth.
By using an expert replacement mechanism, low-scoring expert models are replaced with similar expert models in GPU memory, parameters of high-scoring expert models are prefetched, and a load balancing scheduling algorithm is used to dynamically allocate computing tasks between the GPU and CPU, optimizing the time overlap between computing and data transmission.
It significantly reduced the amount of data transmitted during the inference process, improved the memory hit rate, shortened the generation latency, and enhanced the real-time response quality and interaction smoothness of the edge AI assistant, while maintaining the high accuracy of the model.
Smart Images

Figure CN123117340A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data computing technology, and relates to artificial intelligence and computer systems. Specifically, it relates to a hybrid expert large language model edge scheduling and reasoning method and system based on expert replacement. Background Technology
[0002] With the development of artificial intelligence technology, the demand for large language models in edge devices such as smart homes and autonomous driving is increasing. Hybrid expert architecture, due to its ability to reduce computational load by activating only some parameters, has become a key technology for deploying large models on edge devices.
[0003] However, deploying hybrid expert models on consumer-grade edge hardware faces the significant challenge of limited GPU memory capacity. Since the GPU memory of edge devices typically cannot hold all the expert parameters of the entire model, the inference process requires frequent loading of expert parameters from main memory to GPU memory via the bus, or auxiliary computation using the CPU. Existing solutions mainly include expert prefetching and expert pruning. Expert prefetching attempts to load expert models in advance, but due to limitations in bus bandwidth and prediction accuracy, it is difficult to completely mask transmission latency. Expert pruning improves speed by directly discarding some experts, but often leads to a significant decrease in model inference accuracy. Furthermore, existing scheduling strategies typically treat all experts as equally important, ignoring the differences in the contribution of different experts to the final inference result, resulting in valuable transmission bandwidth being wasted on experts with less impact on the outcome.
[0004] In summary, designing an expert scheduling mechanism that can significantly reduce inference latency while maintaining high model accuracy on memory-constrained edge devices is a pressing technical challenge that needs to be addressed. Summary of the Invention
[0005] Purpose of the invention: This invention aims to propose an edge-side scheduling and inference method and system for hybrid expert (MoE) large language models based on expert replacement. By distinguishing the importance of experts and replacing low-importance experts with similar experts already in the video memory, and prefetching high-scoring experts, this method solves the problem of frequent data transfer caused by limited video memory while ensuring the inference accuracy of the model.
[0006] Technical solution: To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] An edge-side scheduling and inference method for a hybrid expert large language model based on expert substitution includes the following steps:
[0008] Step 1: In the reasoning process of the large language model, for each decoding layer, based on the scoring results of each expert model by the gating network, the set of currently activated expert models is divided into a set of high-scoring expert models and a set of low-scoring expert models.
[0009] Step 2: Determine the low-scoring expert models in the set of low-scoring expert models. If there is a cached expert model in the graphics processor's memory that is similar in function or score to the low-scoring expert model, then use the cached expert model to replace the original activated low-scoring expert model in the calculation; otherwise, retain the original low-scoring expert model.
[0010] Step 3: Predict the high-resolution expert model that may be activated in the next layer. While calculating in the current layer, the parameters of the high-resolution expert model that may be activated in the next layer are preloaded from memory to the video memory of the graphics processor through the bus bandwidth.
[0011] Step 4: For the remaining active expert models that failed to be replaced in Step 2 and were not preloaded into video memory, the load balancing scheduling algorithm is used to determine whether to load the parameters of the remaining active expert models into the graphics processor for calculation or to calculate them directly on the central processing unit, and the final inference results are merged.
[0012] Specifically, in step 1, the currently active set of expert models is divided into a set of high-scoring expert models and a set of low-scoring expert models through the following process:
[0013] Select the score of the (k+1)th expert As a baseline score, a score greater than or equal to Expert models with scores falling within the following ranges are labeled as high-scoring expert models and assigned to the high-scoring expert model set; expert models with scores falling within the following ranges are labeled as low-scoring expert models and assigned to the low-scoring expert model set: ;in, This is used to replace the threshold.
[0014] Specifically, in step 2, cached expert models with similar functionality or scores to the low-scoring expert model are determined using the following formula:
[0015] ,
[0016] in, Scoring of cached expert models.
[0017] Specifically, in step 3, the high-scoring expert model that may be activated in the next layer is predicted through the following process:
[0018] Using the non-shared and shared expert parameters already in the graphics processor's memory, forward computation is performed on the current layer input to generate the predicted hidden state. The hidden state is then processed through the attention mechanism of the next layer to complete the attention computation. After linear transformation and softmax by the next layer's gating network, the predicted routing scores of each expert are obtained, thereby determining the set of high-scoring experts and triggering prefetching.
[0019] Specifically, in step 4, the load balancing scheduling algorithm includes:
[0020] The expert models to be processed are sorted by score. The time required to load the current expert model onto the graphics processing unit (GPU) is compared with the time required to compute it directly on the central processing unit (CPU). The current expert model is dynamically allocated to either the GPU loading queue or the CPU computation queue until all remaining expert models have been processed. The goal of the load balancing scheduling algorithm is:
[0021] ,
[0022] in, The number of experts allocated to the graphics processor's load queue. The time required to load a single expert into the graphics processor, The number of experts allocated to the central processing unit's computing queue. The time required for the central processing unit to compute the calculations for a single expert.
[0023] Furthermore, the time required to load the current expert model onto the graphics processor includes both the graphics processor's computation time and the bus transfer time.
[0024] Furthermore, it also includes score-based cache eviction policies:
[0025] Record the historical activation scores of each expert model over the past n iterations. When there is insufficient video memory, calculate the historical average score of all expert models in video memory and remove the expert with the lowest historical average score from video memory.
[0026] Furthermore, the cache eviction policy temporarily protects the expert model that is about to be accessed in the current layer.
[0027] This invention also provides an edge-side scheduling and inference system for a hybrid expert large language model based on expert substitution, comprising:
[0028] The expert importance partitioning module is used in the reasoning process of a large language model to divide the currently active set of expert models into a set of high-scoring expert models and a set of low-scoring expert models for each decoding layer based on the scoring results of each expert model by the gating network.
[0029] The low-scoring expert replacement module is used to judge the low-scoring expert models in the set of low-scoring expert models. If there is a cached expert model in the graphics processor's video memory that is similar in function or score to the low-scoring expert model, then the cached expert model is used to replace the original activated low-scoring expert model in the calculation; otherwise, the original low-scoring expert model is retained.
[0030] The high-resolution expert prefetching module is used to predict the high-resolution expert models that may be activated in the next layer. While the current layer is being calculated, the parameters of the high-resolution expert models that may be activated in the next layer are preloaded from memory to the graphics processor's video memory through the bus bandwidth.
[0031] The heterogeneous computing load balancing module is used to determine, through a load balancing scheduling algorithm, whether to load the parameters of the remaining active expert models that failed to be replaced in step 2 and were not preloaded into the video memory, to the graphics processor for computation or to compute directly on the central processing unit, and to merge the final inference results.
[0032] Beneficial effects:
[0033] 1. This invention dynamically identifies low-contribution experts in the current inference task through a low-scoring expert replacement mechanism and redirects them to similar experts residing in GPU memory. This avoids the latency caused by loading large-scale parameters from memory to the GPU, significantly reducing the amount of data that needs to be loaded from memory to GPU memory during inference and breaking the limitation of bus bandwidth on inference speed. Combining high-scoring expert cross-layer prediction prefetching and CPU-assisted computation, the GPU performs prediction and prefetching while the CPU performs low-scoring expert computation in parallel. The two work together to achieve time overlap between computation and data transmission, effectively utilizing the bandwidth between computation intervals and further optimizing the utilization of system resources. At the same time, a heterogeneous computing load balancing strategy is combined to achieve minimum end-to-end latency.
[0034] 2. The present invention uses a cache dynamic maintenance strategy based on historical activation scores to ensure that the most valuable high-frequency business experts for inference reside in the video memory, thereby improving the hit rate of experts in the video memory and enabling more calculations to be completed directly in the graphics processor.
[0035] 3. The present invention solves the stuttering problem caused by the memory bottleneck during hybrid expert model inference on the edge, which can improve the real-time response quality of edge AI assistants in scenarios such as smartphones and smart cockpits. While ensuring almost no loss of quality, it significantly shortens the generation latency and improves the interactive smoothness of large edge models under complex tasks. Attached Figure Description
[0036] Figure 1 This is a schematic diagram illustrating the overall process and architecture of the hybrid expert large language model-based scheduling and inference method based on expert replacement provided by the present invention.
[0037] Figure 2 This is a schematic diagram illustrating the principle of the expert replacement mechanism.
[0038] Figure 3 This is a schematic diagram of a pipeline for collaborative computing and load balancing scheduling between the central processing unit and the graphics processing unit. Detailed Implementation
[0039] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.
[0040] Example 1: Figure 1 This diagram illustrates the overall process and architecture of the present invention, showcasing the collaborative working logic of expert classification, replacement, prefetching, and hybrid computation. The present invention proposes a hybrid expert large language model-based on-device scheduling and inference method, the process of which is as follows: Figure 1 As shown, it includes the following steps:
[0041] Step 1, Expert Importance Classification
[0042] Before each layer of reasoning in the large language model begins, the gating network outputs the sorted scores of all activated expert models.
[0043] The system aims to maximize the number of expert hits in the graphics processor's memory, thereby reducing data transfers via the bus. To this end, the optimization objective is defined as maximizing the number of expert hits as described in the following formula:
[0044] ,
[0045] in, This represents the set of expert models already stored in the graphics processor's memory. This represents the set of expert models that are activated in the current layer. This represents the prefetched set of expert models. This represents the set of low-scoring activation expert models. This represents the set of inactive expert models in video memory that can be replaced.
[0046] After obtaining the sorted activated expert model scores from the gating network output, the baseline score is first determined based on the preset parameter k, i.e., the score of the (k+1)th expert is selected. As a baseline score, the system introduces an adjustable replacement threshold coefficient. This coefficient controls the range of low-scoring expert models. Users can adjust this coefficient according to their different needs for accuracy and speed. The larger the coefficient, the more expert models are replaced, and the faster the inference speed. If an expert model's score is greater than or equal to the product of the baseline score and the replacement threshold coefficient plus 1, it is marked as a high-scoring expert model and assigned to the high-scoring expert model set. When multiple experts have the same score, their assignment is determined by their expert ID order to ensure the deterministic nature of the assignment. If an expert model's score falls within the following range:
[0047] ,
[0048] This model is then labeled as a low-scoring expert model and assigned to the low-scoring expert model set. The (k+1)th expert is also included in the low-scoring expert model set. This step of classifying expert models is based on the observation that only a few high-scoring experts have a decisive influence on the output results, while the contribution of low-scoring experts is relatively low.
[0049] Step 2, Replace low-scoring experts
[0050] For the low-scoring expert models after step 1, the system iterates through all currently inactive expert models residing in the graphics processor's memory to find a replacement with a similar score. The replacement's score... To be considered as a candidate set, the following conditions must be met:
[0051] ,
[0052] If a cached expert model that meets the above conditions exists in the video memory, the system will directly use this cached expert model as a substitute expert model to replace the original low-scoring expert model for computation, without reading the parameters of the original low-scoring expert model from memory. Specifically, the index of the expert model in the video memory is pointed to the current computation task, avoiding triggering a page fault to read the original expert data from memory, thereby avoiding the high time cost of loading the original low-scoring expert model from memory to video memory.
[0053] If there is no cached expert model in the video memory that meets the above conditions, then it is still necessary to calculate the low-scoring expert model and participate in the load balancing scheduling in step 4.
[0054] The expert replacement mechanism in this step is as follows: Figure 2 As shown in the figure, it illustrates how to replace a low-scoring expert to be loaded with an idle expert in video memory.
[0055] Step 3, high-scoring experts pre-select:
[0056] The system leverages the powerful parallel computing capabilities of the graphics processing unit (GPU) to predict potentially activated high-scoring expert models in the next layer while simultaneously calculating the expert model for the current layer. This prediction is achieved using parameters from shared expert models and some already loaded expert models. During the current layer's computation, a prefetching operation is initiated, utilizing bus bandwidth to preload parameters from memory into the GPU's video memory. Specifically, the prediction process involves: using the non-shared and shared expert parameters already in the GPU's video memory, performing forward computation on the current layer's input to generate a predicted hidden state; using this hidden state, performing attention computation through the next layer's attention mechanism (key-value cache); and then, through linear transformation and softmax by the next layer's gating network, obtaining the predicted routing scores for each expert, thus determining the set of high-scoring experts and triggering prefetching. Based on the predicted routing scores of each expert, pre-defined rules (such as the TOP-K method) can be used to select prefetched high-scoring experts. Since the GPU is idle at this time, the execution time can be masked by the CPU's computation time, achieving complete overlap between computation and transmission, thus covering the time overhead of data transmission. By prefetching only a small number of high-resolution expert models, the bandwidth during computational gaps can be effectively utilized, reducing bandwidth pressure and improving the effectiveness of prefetching.
[0057] Step 4, Heterogeneous computing load balancing
[0058] For the remaining active experts that were not processed by replacement and were not preloaded into video memory, this invention uses a load balancing scheduling algorithm to dynamically decide, based on the current transmission bandwidth and computing power, whether to load these expert model parameters to the graphics processor for calculation or to calculate them directly on the central processing unit, and merges the final inference results to minimize end-to-end latency.
[0059] The load balancing scheduling algorithm works as follows: the system monitors the bus transfer rate and the CPU's computational load in real time. For tasks not in video memory and not yet replaced, the scheduling algorithm compares the "transfer time plus GPU computation time" with the "CPU's direct computation time," allocating the task to the shorter one. The goal of the scheduling algorithm is to minimize the maximum of these two factors.
[0060] .
[0061] in, The number of experts allocated to the graphics processor's load queue. The time required to load a single expert to the graphics processor (including PCIe transfer time and graphics processor computation time). This is the total latency for the graphics processor to complete all expert loading and calculations. The number of experts allocated to the central processing unit's computing queue. The time required for the central processing unit to compute the calculations for a single expert. This represents the total latency for the central processing unit to complete all expert calculations; min max(·) means minimizing the maximum latency on both sides by optimizing the expert allocation, that is, enabling the GPU path and the CPU path to complete simultaneously, thus achieving optimal parallel scheduling.
[0062] The system employs a two-pointer greedy algorithm, maintaining a list of expert models to be processed. The experts are sorted by their scores, and the system greedily selects the execution method with the lower time cost, dynamically allocating them to either the graphics processor loading queue or the central processing unit computing queue to balance the completion time of both, until all remaining experts are processed, thus reducing waiting vacuum in the entire inference process. Figure 3 The figure shows a schematic diagram of the pipeline for collaborative computing and load balancing scheduling in this step. As shown, this invention takes into account the time overlap of graphics processor computing, central processing unit computing and bus transmission, and ensures the compactness of the processing pipeline through algorithms.
[0063] Experimental results show that, under three hardware configurations (RTX 3080Ti / 12GB, RTX 4060Ti / 16GB, NVIDIA A6000 / 48GB) and various workloads (GaoKao, WiC, TriviaQA, Race-mid, GSM8K, MT-Bench, MMLU, HumanEval), compared with the best baseline, this method (SMoE) reduces the average generation latency per token (TPOT) by 24% with a batch size of 1 and by 35% with a batch size of 3 (up to 48% reduction under the A6000 configuration); the overall inference latency is reduced by an average of 23.7%; the graphics processor expert cache hit rate is improved by at least 65%; the first token latency (TTFT) is reduced by an average of 11%; the high-scoring expert prefetch accuracy is approximately 82%; and the accuracy loss in each benchmark is negligible when the replacement threshold alpha <= 0.35, and in some scenarios, the accuracy is even better than the direct inference results of the original model.
[0064] As an improvement, this invention also employs a score-based cache eviction strategy to determine which experts to remove when video memory is insufficient. Specifically, the system maintains a record table storing the historical activation scores of each expert over the most recent n time steps. When video memory is full and a new expert needs to be loaded, the system scans this record table and calculates the average historical score of all experts in video memory. For expert models, their eviction scores are... The calculation formula is as follows:
[0065] .
[0066] In the above formula, the numerator is the sum of the historical activation scores at each step within the observation window, and the denominator is the actual number of iterations within the window. A lower score indicates a smaller contribution of the expert to the model results. The expert with the lowest score is marked as a victim and removed from memory, thus ensuring that the most valuable experts for inference are always retained in memory. To prevent the erroneous removal of experts about to be used, experts selected to participate in the computation at the current inference layer are given a temporary protection status and a temporary lock flag is set.
[0067] Through the above-described embodiments, this invention enables the smooth execution of hybrid expert large language models with a parameter count far exceeding the available video memory on edge devices (such as personal computers and embedded systems) with limited video memory resources. Compared to traditional full loading or simple pruning methods, this invention significantly improves the text generation rate while maintaining high accuracy.
[0068] Based on the same technical concept as the aforementioned hybrid expert large language model edge scheduling and inference method based on expert substitution, this invention also provides an edge scheduling and inference system based on expert substitution for hybrid expert large language models, comprising:
[0069] The expert importance partitioning module is used in the reasoning process of a large language model to divide the currently active set of expert models into a set of high-scoring expert models and a set of low-scoring expert models for each decoding layer based on the scoring results of each expert model by the gating network.
[0070] The low-scoring expert replacement module is used to judge the low-scoring expert models in the set of low-scoring expert models. If there is a cached expert model in the graphics processor's video memory that is similar in function or score to the low-scoring expert model, then the cached expert model is used to replace the original activated low-scoring expert model in the calculation; otherwise, the original low-scoring expert model is retained.
[0071] The high-resolution expert prefetching module is used to predict the high-resolution expert models that may be activated in the next layer. While the current layer is being computed, the parameters of the high-resolution expert models that may be activated in the next layer are preloaded from memory into the graphics processor's video memory through the bus bandwidth.
[0072] The heterogeneous computing load balancing module is used to determine, through a load balancing scheduling algorithm, whether to load the parameters of the remaining active expert models that failed to be replaced in step 2 and were not preloaded into the video memory, to the graphics processor for computation or to compute directly on the central processing unit, and to merge the final inference results.
[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0074] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0077] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A hybrid expert large language model scheduling and reasoning method based on expert substitution, characterized in that, Includes the following steps: Step 1: In the reasoning process of the large language model, for each decoding layer, based on the scoring results of each expert model by the gating network, the set of currently activated expert models is divided into a set of high-scoring expert models and a set of low-scoring expert models. Step 2: Determine the low-scoring expert models in the set of low-scoring expert models. If there is a cached expert model in the graphics processor's memory that is similar in function or score to the low-scoring expert model, then use the cached expert model to replace the original activated low-scoring expert model in the calculation; otherwise, retain the original low-scoring expert model. Step 3: Predict the high-resolution expert model that may be activated in the next layer. While calculating in the current layer, the parameters of the high-resolution expert model that may be activated in the next layer are preloaded from memory to the video memory of the graphics processor through the bus bandwidth. Step 4: For the remaining active expert models that failed to be replaced in Step 2 and were not preloaded into video memory, the load balancing scheduling algorithm is used to determine whether to load the parameters of the remaining active expert models into the graphics processor for calculation or to calculate them directly on the central processing unit, and the final inference results are merged.
2. The end-side scheduling and reasoning method for hybrid expert large language models based on expert substitution according to claim 1, characterized in that, In step 1, the currently active set of expert models is divided into a set of high-scoring expert models and a set of low-scoring expert models through the following process: Select the score of the (k+1)th expert As a baseline score, a score greater than or equal to Expert models with scores falling within the following ranges are labeled as high-scoring expert models and assigned to the high-scoring expert model set; expert models with scores falling within the following ranges are labeled as low-scoring expert models and assigned to the low-scoring expert model set: ;in, This is used to replace the threshold.
3. The end-side scheduling and reasoning method for a hybrid expert large language model based on expert substitution according to claim 1, characterized in that, In step 2, cached expert models with similar functionality or scores to the low-scoring expert model are determined using the following formula: , in, Scoring of cached expert models.
4. The end-side scheduling and inference method for a hybrid expert large language model based on expert substitution according to claim 1, characterized in that, In step 3, the high-scoring expert model that may be activated in the next layer is predicted through the following process: Using the non-shared and shared expert parameters already in the graphics processor's memory, perform forward computation on the current layer input to generate the predicted hidden state; then use the hidden state to complete the attention computation through the attention mechanism of the next layer. After linear transformation and softmax by the next layer of gating network, the predicted route scores of each expert are obtained, thereby determining the set of high-scoring experts and triggering prefetching.
5. The end-side scheduling and reasoning method for a hybrid expert large language model based on expert substitution according to claim 1, characterized in that, In step 4, the load balancing scheduling algorithm specifically includes: The expert models to be processed are sorted by score. The time required to load the current expert model onto the graphics processing unit (GPU) is compared with the time required to compute it directly on the central processing unit (CPU). The current expert model is dynamically allocated to either the GPU loading queue or the CPU computation queue until all remaining expert models have been processed. The goal of the load balancing scheduling algorithm is: , in, The number of experts allocated to the graphics processor's load queue. The time required to load a single expert into the graphics processor, The number of experts allocated to the central processing unit's computing queue. The time required for the central processing unit to compute the calculations for a single expert.
6. The end-side scheduling and reasoning method for a hybrid expert large language model based on expert substitution according to claim 5, characterized in that, The time required to load the current expert model into the graphics processor includes both the graphics processor's computation time and the bus transfer time.
7. The end-side scheduling and reasoning method for a hybrid expert large language model based on expert substitution according to claim 1, characterized in that, It also includes a score-based cache eviction policy: Record the historical activation scores of each expert model over the past n iterations. When there is insufficient video memory, calculate the historical average score of all expert models in video memory and remove the expert with the lowest historical average score from video memory.
8. The end-side scheduling and reasoning method for a hybrid expert large language model based on expert substitution according to claim 7, characterized in that, In a score-based cache eviction policy, the expert model that is about to be accessed in the current layer is temporarily protected.
9. A hybrid expert large language model edge scheduling and reasoning system based on expert substitution, characterized in that, include: The expert importance partitioning module is used in the reasoning process of a large language model to divide the currently active set of expert models into a set of high-scoring expert models and a set of low-scoring expert models for each decoding layer based on the scoring results of each expert model by the gating network. The low-scoring expert replacement module is used to judge the low-scoring expert models in the set of low-scoring expert models. If there is a cached expert model in the graphics processor's video memory that is similar in function or score to the low-scoring expert model, then the cached expert model is used to replace the original activated low-scoring expert model in the calculation; otherwise, the original low-scoring expert model is retained. The high-resolution expert prefetching module is used to predict the high-resolution expert models that may be activated in the next layer. While the current layer is being calculated, the parameters of the high-resolution expert models that may be activated in the next layer are preloaded from memory to the graphics processor's video memory through the bus bandwidth. The heterogeneous computing load balancing module is used to determine, through a load balancing scheduling algorithm, whether to load the parameters of the remaining active expert models that failed to be replaced in step 2 and were not preloaded into the video memory, to the graphics processor for computation or to compute directly on the central processing unit, and to merge the final inference results.
