Heterogeneous hybrid expert large model inference optimization method and system based on dynamic expert scheduling

Abstract

The method and system for heterogeneous hybrid expert large model inference optimization based on dynamic expert scheduling include: a state initialization unit determines a target expert set remaining in a first computing unit in a global range based on a set global hardware resource budget and historical activation features of experts of a whole model, and generates a global expert scheduling state; a concurrent execution unit, when performing forward inference of a model, dispatches an expert computing task that hits a resident to a main execution path of the first computing unit according to the global expert scheduling state, and dispatches an expert computing task that does not hit a resident to an independent execution path of a second computing unit; and outputs a result after both are calculated; an adaptive updating unit re-determines the target expert set of the first computing unit; and updates the global expert scheduling state and the corresponding expert weight on the first computing unit relied on by the heterogeneous concurrent execution step in place without interrupting the continuity of the main execution path calculation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and deep learning technology, and more specifically, to a heterogeneous hybrid expert large model inference optimization method and system based on dynamic expert scheduling. Background Technology

[0002] In recent years, large-scale language models (LMs) have demonstrated outstanding capabilities in tasks such as natural language processing, intelligent question answering, and code generation, driving the continuous growth of model parameter size. Mixture-of-Experts (MoE) architectures, due to their sparse activation characteristics—large parameter size but low actual computational cost—have become the mainstream technical approach for scaling up models. Typical MoE models, such as DeepSeek-R1, Qwen3-MoE, and MiniMax-M2, activate only a small number of experts in each forward inference iteration. For example, the MiniMax-M2 model has a total of 200 billion parameters, but only uses about 10 billion parameters in a single forward propagation. However, as the scale of MoE models continues to expand, their inference deployment on consumer-grade hardware faces significant challenges.

[0003] First, there's the bottleneck of GPU memory capacity. The total number of parameters in a MoE model often far exceeds the total memory of a single or multiple consumer-grade GPUs. Therefore, some expert weights must be offloaded to CPU memory, introducing CPU-GPU heterogeneous collaboration. However, in heterogeneous inference scenarios, DDR memory bandwidth and the bandwidth of the high-speed PCIe bus interconnect for peripheral components are orders of magnitude lower than GPU memory bandwidth. Frequent data transfers can lead to significant inference latency.

[0004] Secondly, existing technologies for this type of heterogeneous inference typically employ either a coarse static uniform placement strategy or an on-demand real-time loading strategy. The static uniform placement strategy usually assumes a uniform distribution of expert importance across layers, selecting the same number of experts at each layer to reside on the GPU according to fixed numbers. However, in reality, the activation frequency of different experts in the MoE model exhibits a significant polarization, with substantial differences in activation patterns across different layers and inputs. This results in a large amount of limited GPU memory being allocated to rarely activated, less popular experts, while truly popular experts remain on the CPU, leading to extremely low GPU resource utilization and expert hit rate. Furthermore, this strategy lacks runtime adaptability and cannot respond to changes in workload, such as changes in user request types.

[0005] Finally, regarding the efficiency of heterogeneous parallel execution, existing heterogeneous inference frameworks mostly adopt a CPU-GPU single-stream serial execution approach. This approach completely exposes the fundamental overhead of CPU-side data transfer, task submission, and synchronization waiting to the critical path of GPU computation, making it impossible for GPU computation to mask it. Even when all experts at certain layers reside on the GPU, the fundamental CPU overhead still exists, resulting in the performance of heterogeneous frameworks being inferior to pure GPU benchmark solutions.

[0006] Therefore, the existing technology has problems and needs further improvement and development. Summary of the Invention

[0007] (I) Purpose of the invention: In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a heterogeneous hybrid expert large model reasoning optimization method and system based on dynamic expert scheduling.

[0008] (II) Technical Solution: To address the aforementioned technical problems, this technical solution provides a heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling, comprising the following steps:

[0009] Step S1: Based on the set global hardware resource budget and the historical activation characteristics of each expert in the whole model, determine the target expert set that will preferentially reside in the first computing unit in the global scope, and generate a global expert scheduling state for global overall planning of expert computing task routing accordingly.

[0010] Step S2: When performing forward inference of the model, according to the indication of the currently effective global expert scheduling state, the expert calculation tasks that have been successfully assigned to the main execution path of the first computing unit are dispatched, while the expert calculation tasks that have not been successfully assigned to the independent execution path of the second computing unit are dispatched; wherein, the main execution path and the independent execution path are processed asynchronously and concurrently in parallel, and the results are merged and output after both have been calculated;

[0011] Step S3: During the execution of step S2, the actual activation features of the inference request currently being processed by the large model are extracted as a feedback signal to trigger dynamic evaluation, so as to redetermine the target expert set of the first computing unit; and without interrupting the computational continuity of the main execution path, the global expert scheduling state on which step S2 depends and the corresponding expert weights on the first computing unit are updated in place, so that subsequent forward inference tasks can continue to be executed asynchronously and concurrently based on the updated global expert scheduling state.

[0012] Specifically, the global expert scheduling state is a multidimensional Boolean tensor, the dimension of which is determined by the number of layers in the large model and the number of experts in each layer.

[0013] Step S1 further includes: based on the global expert scheduling state, establishing an index mapping table from the logical expert number of the large model to the physical slot number of the continuous weight area within the first computing unit.

[0014] The specific methods for determining the target expert set that will preferentially reside in the first computing unit include:

[0015] Obtain the expert activation distribution data generated during the offline calibration phase;

[0016] Within the entire model, all experts are globally sorted from high to low according to the activation frequency in the activation distribution data.

[0017] The experts with the highest activation frequency are selected in sequence and added to the target expert set until the upper limit of the global hardware resource budget is reached.

[0018] The first computing unit is a graphics processing unit (GPU) containing high-bandwidth video memory, and the second computing unit is a central processing unit (CPU) that works in conjunction with main memory.

[0019] In step S2, the global expert scheduling state is allocated in fixed memory and shared consistently across computing devices between the first inference engine responsible for scheduling the first computing unit and the second inference engine responsible for scheduling the second computing unit.

[0020] Specifically, the asynchronous concurrent parallel processing of the main execution path and the independent execution path includes:

[0021] On the main execution path of the first computing unit, the input tensor of the current layer is copied to the temporary buffer;

[0022] The independent execution path of the second computing unit detects the completion of the copy through event synchronization, reads the immutable data copy in the temporary buffer, and independently executes the expert inference calculation of the second computing unit; while the main execution path executes the expert inference calculation of the first computing unit in parallel on the original input tensor.

[0023] After the main execution path completes its calculation, it waits for the independent execution path to complete its calculation via event synchronization in order to merge the two calculation results.

[0024] The triggering conditions for extracting the actual activation features currently being processed by the large model as a feedback signal to trigger dynamic evaluation include any one or more of the following combinations:

[0025] During the filling phase of model inference, the number of tags in the current batch reaches the first preset threshold.

[0026] Since the last update of the global expert scheduling status, the continuous running time of the inference service has reached the second preset period;

[0027] Within the preset operating window, the actual hit rate of the target expert set resident by the first computing unit is lower than the third preset threshold.

[0028] Since the last update of the global expert scheduling status, the cumulative number of inference requests processed has reached the fourth preset threshold.

[0029] Specifically, when dynamic evaluation is triggered during the filling phase of the model inference, the forward propagation of the current batch is completed by constructing a temporary full expert layer rollback mode on the first computing unit, so as to fully obtain the actual expert activation distribution of the current batch in the gating network output, and use it as the basis for redetermining the target expert set.

[0030] The specific method for updating the global expert scheduling state and the corresponding expert weights on the first computing unit in-situ includes:

[0031] Copy the weight data corresponding to the newly selected target expert set to the continuous weight area of ​​the first calculation unit;

[0032] The global expert scheduling state, its copy on the first computing unit side, and the index mapping table are overwritten in place.

[0033] The in-situ update process maintains the global expert scheduling state and its copy, as well as the memory address of the index mapping table, unchanged.

[0034] The specific methods for determining the target expert set that will preferentially reside in the first computing unit include any of the following:

[0035] Weighted frequency strategy: When calculating the original activation counts of each expert, layer weights or positional decay factors are introduced to give higher allocation priority to layers closer to the output layer or with higher computational overhead. After obtaining the weighted activation score, experts are selected from high to low according to the weighted activation score to join the target expert set; or,

[0036] Activation time weighting strategy: Record the timestamp or request number of each expert activation event, calculate the time distance between each activation event and the current time, and calculate the corresponding decay weight according to a preset decay function, wherein the larger the time distance, the smaller the decay weight. The preset decay function includes an exponential decay function or a piecewise decay function. The decay weights of all activation events for the same expert are weighted and summed to obtain the time-decayed activation score, and experts are selected from high to low activation scores to join the target expert set; or...

[0037] Sliding time window statistical strategy: Maintain a time window of preset length. When a new inference request or batch is routed, the activation count of each expert is added to the statistical value of the time window. When a historical inference request or batch exceeds the range of the time window, the corresponding activation count is deducted from the statistical value. Experts are selected from high to low based on the cumulative activation count of each expert within the current time window and added to the target expert set.

[0038] In a distributed deployment scenario that includes multiple of the first computing units:

[0039] The computational load of the large model forward inference is distributed to multiple first computing units in parallel execution through tensor parallelism.

[0040] The global expert scheduling state is generated uniformly by the master process node and synchronized to all other process nodes participating in tensor parallelism through set communication, so as to ensure that all processes have a consistent view of the expert's residence location.

[0041] A heterogeneous hybrid expert large model inference optimization system based on dynamic expert scheduling includes:

[0042] The state initialization unit is used to determine the set of target experts who should reside in the first computing unit in the global scope based on the set global hardware resource budget and the historical activation characteristics of each expert in the whole model, and generate a global expert scheduling state for global overall planning of expert computing task routing.

[0043] The concurrent execution unit is used to dispatch expert computation tasks that have been successfully executed to the main execution path of the first computing unit, and dispatch expert computation tasks that have not been successfully executed to the independent execution path of the second computing unit, according to the indication of the currently effective global expert scheduling state, during the forward inference of the execution model; wherein the main execution path and the independent execution path are processed in parallel through an asynchronous concurrency mechanism, and the results are merged and output after both have been computed;

[0044] An adaptive update unit is used to extract the actual activation features of the inference request currently being processed by the large model as a feedback signal to trigger dynamic evaluation during the operation of the concurrent execution unit, so as to redetermine the target expert set of the first computing unit; and without interrupting the computational continuity of the main execution path, update the global expert scheduling state on which the concurrent execution unit depends and the corresponding expert weights on the first computing unit in place, so that subsequent forward inference tasks can continue to be executed asynchronously and concurrently based on the updated global expert scheduling state.

[0045] (III) Beneficial effects: This invention provides a heterogeneous hybrid expert large model inference optimization method and system based on dynamic expert scheduling. By coordinating global allocation, dynamic updating and asynchronous concurrency, it breaks through the hierarchical limitation, significantly improves the GPU expert hit rate, shields basic overhead, restores the maximum inference throughput, achieves runtime adaptation, perfectly copes with distributed off-workload, and achieves seamless state updates with zero overhead. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the architecture of the heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling of the present invention;

[0047] Figure 2 This is a schematic diagram of the heterogeneous hybrid expert large model inference optimization system based on dynamic expert scheduling, as described in this invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to preferred embodiments. More details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can obviously be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual application situations without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0049] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that these drawings are for illustrative purposes only and are not drawn to scale, and should not be construed as limiting the actual scope of protection of the present invention.

[0050] A heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling is applied to, for example, Figure 2 The figure shows a heterogeneous hybrid expert large model inference optimization system based on dynamic expert scheduling, including a state initialization unit, a concurrent execution unit, and an adaptive update unit.

[0051] A heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling, such as Figure 1 As shown, the specific steps include:

[0052] Step S1: Based on the set global hardware resource budget and the historical activation characteristics of each expert in the whole model, the state initialization unit determines the target expert set that should reside in the first computing unit in the global scope, and generates a global expert scheduling state for global overall planning of expert computing task routing.

[0053] In step S2, when the concurrent execution unit performs forward inference of the model, it dispatches the expert computation tasks that have been successfully executed to the main execution path of the first computing unit according to the indication of the currently effective global expert scheduling state, and dispatches the expert computation tasks that have not been successfully executed to the independent execution path of the second computing unit; wherein, the main execution path and the independent execution path are processed asynchronously and concurrently, and the results are merged and output after both have been computed;

[0054] In step S3, during the operation of step S2, the adaptive update unit extracts the actual activation features of the inference request currently being processed by the large model as a feedback signal to trigger dynamic evaluation, so as to redetermine the target expert set of the first computing unit; and without interrupting the computational continuity of the main execution path, updates the global expert scheduling state on which step S2 depends and the corresponding expert weights on the first computing unit in place, so that subsequent forward inference tasks can continue to be executed asynchronously and concurrently based on the updated global expert scheduling state.

[0055] The preferred large-scale model is the MoE (Mixture-of-Experts) model, which inserts multiple expert sub-networks (Experts) into the conventional deep learning architecture Transformer or other deep network layers based on self-attention mechanisms. A dynamic routing gating mechanism is introduced to automatically select a few experts for activation and computation based on input features. The MoE model invokes a subset of experts for each input sample or position, allowing the parameter scale to grow linearly while the computational cost increases only at a constant level, enabling model designs with hundreds of billions or even trillions of parameters. During inference, the MoE-based model generates different expert scores, allowing independent monitoring and tuning of the weights and routing strategies of each expert. Simultaneously, dynamic routing supports sparse updates, expert replacements, and expert expansions, providing finer-grained model evolution and maintainability.

[0056] The global hardware resource budget refers to the upper limit of the total number of resident expert weights that can be supported by the high-speed video memory space reserved for the first computing unit in a heterogeneous inference platform.

[0057] The historical activation features of each expert in the whole model refer to the prior distribution features of the probability and activation density of each expert subnetwork being selected by the gating network in the initialization phase of model deployment, which are obtained through offline calibration and represent the whole model scope.

[0058] The main execution path is established within the execution context of the first computing unit and relies on the main computing stream, such as the main CUDA stream. Its function is to execute tensor kernel calculations for high-frequency experts residing in GPU memory that have been hit, and to coordinate and control the temporal evolution of the entire forward inference process. The independent execution path is completely separated from the main computing stream, established within the control context of the second computing unit, and relies on independent auxiliary streams, such as independent CPU computing threads or auxiliary CUDA streams. It is a parallel execution path used to handle the computational tasks of less popular experts that have not been hit, including submitting input tensors from main memory to CPU expert operators for inference, and returning the results after computation on the CPU. The main execution path and the independent execution path achieve data isolation and temporal coordination through temporary buffers and event synchronization. This allows the time-consuming CPU data preparation, task submission, and synchronization waiting overhead on the independent execution path to be largely masked by the high-concurrency GPU expert computations on the main execution path, thereby achieving efficient asynchronous concurrency of computing resources on both sides.

[0059] The first computing unit is typically a graphics processing unit (GPU) equipped with high-bandwidth video memory (HBM) or GDDR, while the second computing unit is a central processing unit (CPU) that works in conjunction with main memory, namely large-capacity dynamic random access memory (DRAM). The two are connected via a high-speed serial interconnect bus.

[0060] GPU, or Graphics Processing Unit, is a highly parallel computing processor originally used for graphics rendering but now widely used in deep learning training and inference. GPUs have thousands of computing cores and high-bandwidth video memory (HBM or GDDR), enabling them to efficiently perform large-scale matrix operations and tensor operations in a SIMT (Single Instruction, Multiple Threads) parallel manner, making them the primary hardware platform for large language model inference.

[0061] CPU, or Central Processing Unit, is a general-purpose processor in a computer, possessing flexible control logic and multi-core parallel capabilities. In heterogeneous inference scenarios of large MoE models, the CPU, in conjunction with large-capacity DRAM memory, can handle expert computation tasks that overflow when GPU memory is insufficient. Its advantages in multi-core parallelism and memory capacity make it an important supplementary computing resource to the GPU.

[0062] DRAM, or Dynamic Random Access Memory, is the most common type of main memory in computer systems. Compared to GPU memory, DRAM has a larger capacity but lower bandwidth. In the heterogeneous inference architecture of this invention, DRAM is used to store expert weights on the CPU side, and its capacity advantage enables the deployment of ultra-large-scale MoE models on consumer-grade hardware.

[0063] The state initialization unit includes an expert activation statistics module and a first expert placement decision module. The expert activation statistics module records the historical activation characteristics of each expert in the entire model. Specifically, the expert activation statistics module records the activation count or activation frequency of each expert based on the output of the gating network.

[0064] The global expert scheduling state is specifically represented as a multi-dimensional Boolean tensor, which serves as a global mask. Its dimensions are determined by the number of layers in the large model and the number of experts in each layer, specifically represented as [num_layers, num_experts]. The selected expert's position is written as True, and the remaining positions are written as False. This masking mechanism decouples the expert selection strategy from the underlying scheduling execution logic, allowing the same execution path to support multiple placement strategies, providing a unified interface for flexible switching and dynamic adjustment.

[0065] For example, when `num_layers` is 3, `num_experts` is 4, and `gpu_expert_budget` is 5, the global mask has 3 rows and 4 columns, allowing a maximum of 5 elements to be True, with the rest being False. If the GPU-resident expert selected by the initialization strategy is the first expert in layer 0, the third expert in layer 0, the 0th expert in layer 1, the first expert in layer 2, and the second expert in layer 2, then the global mask can be represented as [[False, True, False, True], [True, False, False, False], [False, True, True, False]]. During inference, if the gating network selects the first expert in layer 2, the system reads the corresponding element in the mask as True and dispatches the expert request to the GPU execution path; if the gating network selects the third expert in layer 1, the system reads the corresponding element in the mask as False and dispatches the expert request to the CPU execution path or other non-GPU-resident paths.

[0066] The current placement of each expert in each layer of the large model is a scheduling state shared by the state initialization unit, concurrent execution unit, and adaptive update unit.

[0067] To ensure that the logical expert IDs output by the gating network can correctly access the expert weights stored compactly on the GPU side, the mapping module further establishes an expert weight mapping relationship, i.e., an index mapping table, after generating a global mask. This table addresses two types of correspondences: first, the correspondence between logical expert IDs and the original model weights; and second, the correspondence between logical expert IDs and actual slots in the GPU's contiguous weight region. The former determines which set of model weights each expert ID corresponds to, while the latter helps quickly locate the actual weights involved in the computation when the GPU only stores a portion of the expert weights.

[0068] The mapping module determines the correspondence between logical expert IDs and original weights during model loading based on the model structure and checkpoint weight layout. For the e-th expert in layer l, its logical expert ID can be represented as (l, e), and the corresponding weight is the weight of the expert with ID e in layer l of the MoE, including the expert's gated projection, up projection, down projection, and corresponding bias or quantization parameters. The mapping module traverses the expert modules layer by layer, recording the correspondence between each logical expert ID and its original weight tensor or CPU-side weight storage location. The expert ID output by the gating network during inference is still the logical expert ID, so the meaning of the expert ID remains consistent with the original model.

[0069] Step S1 further includes: the mapping module establishes an index mapping table based on the global expert scheduling state, which maps the logical expert number of the large model to the physical slot number of the continuous weight area in the first computing unit.

[0070] The determination of the target expert set to preferentially reside in the first computing unit is executed by the first expert placement decision module. The specific method includes: obtaining expert activation distribution data generated during the offline calibration phase; globally sorting all experts from high to low activation frequency in the activation distribution data across the entire model; and sequentially selecting the expert with the highest activation frequency to add to the target expert set until the upper limit of the global hardware resource budget is reached.

[0071] To establish the correspondence between logical expert IDs and physical slot numbers in the GPU continuous weight region, the mapping module builds a GPU weight index mapping table based on the global mask. For experts whose global mask value is True, the mapping module first finds the corresponding original weight based on their logical expert ID, and then copies the complete weight of that expert to the GPU continuous weight region of the layer. Since the GPU only stores the weights of the selected experts, the physical slot number of an expert in the GPU continuous weight region is usually different from its original expert ID. Therefore, the mapping module maintains an index mapping table gpu_expert_index. The index mapping table can have the same shape as the global mask [num_layers, num_experts], where gpu_expert_index[l][e] represents the physical slot number of the e-th logical expert in the l-th layer in the GPU continuous weight region of that layer; if the expert is not resident in the GPU, the corresponding position is recorded as an invalid value of -1. During inference, the system first determines whether the target expert resides on the GPU based on the global mask. If the expert resides on the GPU, it continues to read gpu_expert_index to obtain the physical slot number in the actual GPU continuous weight area and calls the weight in the physical slot number in that continuous weight area to complete the expert calculation. If the expert does not reside on the GPU, the expert request is handed over to the CPU execution path for processing.

[0072] For example, suppose the model has two MoE layers, each containing four experts, and the global mask is [[False,True, False, True], [True, False, False, False]]. This mask indicates that the first expert in layer 0, the third expert in layer 0, and the zeroth expert in layer 1 reside on the GPU. The mapping module determines the original weights W0,1 corresponding to the first expert in layer 0, W0,3 corresponding to the third expert in layer 0, and W1,0 corresponding to the zeroth expert in layer 1 based on the original weight layout of the model. Then, W0,1 is placed in physical slot 0 of the continuous weight region of the GPU in layer 0, W0,3 is placed in physical slot 1 of the continuous weight region, and W1,0 is placed in physical slot 0 of the continuous weight region of the GPU in layer 1. At this time, the mapping table can be represented as [[-1, 0, -1, 1], [0, -1, -1, -1]]. When the gating network selects the 3rd expert at layer 0, it reads the mask value as True and further reads gpu_expert_index[0][3] as 1, thereby calling W0,3 in physical slot 1 of the GPU continuous weight region at layer 0; when the gating network selects the 2nd expert at layer 1, it reads the mask value as False, and the corresponding position in the mapping table is -1, so the expert does not go through the physical slot of the GPU continuous weight region, but enters the CPU execution path.

[0073] The concurrent execution unit includes an execution module and a synchronization module. Based on the currently active global expert scheduling status, the execution module dispatches expert computation tasks that have been successfully scheduled to the main execution path of the first computation unit, while dispatching expert computation tasks that have not been scheduled to the independent execution path of the second computation unit. After the computation is completed on the main execution path and the independent execution path, the synchronization module merges and outputs the results.

[0074] Obtaining expert activation distribution data generated during the offline calibration phase refers to pre-constructing a calibration workload dataset that can represent the characteristics of real business requests before the MoE large model is officially run and online inference services are provided. This could be a standard industry benchmark set or a set of user prompt word requests extracted from historical inference services and anonymized.

[0075] To mask the basic overhead of the second computing unit, i.e. the CPU side, this invention separates CPU expert-related operations, including input handling, CPU-side inference submission and result return, from the main CUDA stream and arranges them to be executed on an independent CPU stream, while the main stream focuses on the mask remapping and computation of GPU experts.

[0076] In terms of execution flow, the mainstream approach first copies the input to a temporary buffer cached in batches. The CPU stream waits for this copy to complete via CUDA events before independently proceeding with CPU-side computation, operating in complete parallel with the GPU expert computation on the mainstream platform. Since the GPU expert may modify the input tensor in-place, while the CPU stream reads an immutable copy from the temporary buffer, data contention between the two streams is naturally avoided.

[0077] After completing GPU computation, the mainstream approach waits for the CPU stream to finish via CUDA events before merging and outputting the two results. Through dual-stream parallelism and event synchronization, the CPU overhead is largely hidden within the GPU computation process, allowing the heterogeneous framework to recover to near-pure GPU baseline performance levels when most experts at a certain level are on the GPU.

[0078] This invention enables flexible expression of expert positions and runtime adaptation of placement schemes. Researchers do not need to repeatedly modify the underlying code for different models; they can achieve near-pure GPU inference capabilities for large MoE models with just a few configuration parameters, using a hardware combination of consumer-grade GPUs and conventional CPUs / DRAM.

[0079] CUDA, or Compute Unified Device Architecture, is a general-purpose computing programming platform and API provided by NVIDIA for GPUs. It allows developers to directly utilize the parallel computing capabilities of GPUs to perform general-purpose computing tasks. The GPU-side expert computation, flow management, and graph capture operations in this invention are all implemented based on the CUDA platform.

[0080] In step S2, the entire decision-making and dispatch logic for whether a resident expert is hit or not depends on the global expert scheduling state and index mapping table generated in step S1. The global expert scheduling state and index mapping table are created in fixed memory and are consistently shared across devices between the inference engine responsible for scheduling the first computing unit and the inference engine responsible for scheduling the second computing unit.

[0081] During forward inference, the gating network of the current layer calculates the input token and selects the k highest-scoring target experts. It iterates through these k experts, directly retrieving the value of the corresponding coordinate [l][e] in the global expert scheduling state Boolean tensor based on the current expert's layer number l and expert number e. If the Boolean value at that position is True, it is considered a successful entry. Furthermore, it reads the index mapping table gpu_expert_index[l][e] to obtain the pre-allocated physical slot number of the logical expert in the continuous weight region of the first computation unit. Then, it calls the continuous weight tensor stored in that physical slot to dispatch the computation task to the main execution path, completing high-speed GPU expert inference directly within the GPU memory. The gating network selects the sparse expert combination required for the current token, retrieving the value of the position [l][e] in the global expert scheduling state Boolean tensor based on the target expert's layer number l and expert number e. If the Boolean value at that position is False, it is considered a failed entry. In this case, the value recorded at the corresponding position in the index mapping table is invalid.

[0082] In step S2, the synchronization module allocates the global expert scheduling state to a fixed memory and performs cross-computing device consistency sharing between the first inference engine responsible for scheduling the first computing unit and the second inference engine responsible for scheduling the second computing unit.

[0083] The asynchronous concurrent parallel processing of the main execution path and the independent execution path specifically includes: on the main execution path of the first computing unit, copying the input tensor of the current layer to a temporary buffer; the independent execution path of the second computing unit detects the completion of the copy through event synchronization, reads the immutable data copy in the temporary buffer, and independently executes the expert inference calculation of the second computing unit; at the same time, the main execution path executes the expert inference calculation of the first computing unit in parallel on the original input tensor; after completing the calculation, the main execution path waits for the independent execution path to complete the calculation through event synchronization, so as to merge the two calculation results.

[0084] The adaptive update unit extracts the actual activation features of the current workload, i.e., the inference requests currently being processed by the large model, as a feedback signal to trigger dynamic evaluation. The triggering conditions include any one or more of the following: during the model inference filling phase, the number of labels in the current batch reaches a first preset threshold; the continuous running time of the inference service reaches a second preset period since the last update of the global expert scheduling state; the actual hit rate of the resident target expert set in the first computing unit is lower than a third preset threshold within a preset running window; and the cumulative number of inference requests processed since the last update of the global expert scheduling state reaches a fourth preset threshold. The preset running window refers to a statistical measurement interval used for adaptive, sliding observation of the target expert resident hit rate on the first computing unit during the operation of the heterogeneous inference service.

[0085] When dynamic evaluation is triggered during the filling phase of the model inference, the forward propagation of the current batch is completed by constructing a fallback mode of a temporary full expert layer on the first computing unit, so as to fully obtain the actual expert activation distribution of the current batch in the gating network output, and use it as the basis for redetermining the target expert set.

[0086] The in-situ update of the corresponding expert weights on the first computing unit refers to the dynamic hot replacement of the resident target expert weights within the first computing unit based on feedback from actual activation characteristics during runtime, without altering the physical topology of the video memory. By continuously copying the new high-frequency expert weights evaluated in the current batch to the continuous weight area of ​​the video memory of the first computing unit via a high-speed bus, replacing the original low-frequency activated expert weights, the limited storage resources of the first computing unit are ensured to always serve the current actual hot workload, significantly improving the hardware hit rate of experts. Secondly, the weight update process is performed using an in-situ numerical overwrite method. Before and after the update, the continuous video memory base addresses, spatial boundaries, and tensor underlying physical pointers of each physical slot storing expert weights on the first computing unit remain completely fixed and absolutely unchanged. This absolute stability of the video memory physical addresses makes the dynamic replacement operation of experts completely transparent to the static computation graph already captured and solidified during the decoding stage. Subsequent forward inference tasks do not need to pause the pipeline or rebuild the computation graph due to changes in expert weights, achieving adaptive adjustment of expert distribution with zero scheduling overhead, ensuring high throughput and computational continuity of the entire heterogeneous inference network under dynamic loads.

[0087] The in-situ update of the global expert scheduling state and the corresponding expert weights on the first computing unit specifically includes: copying the weight data corresponding to the newly selected target expert set to the contiguous weight area of ​​the first computing unit; in-situ numerical overwriting of the global expert scheduling state, its copy on the first computing unit side, and the index mapping table; and maintaining the memory addresses of the global expert scheduling state, its copy, and the index mapping table unchanged during the in-situ update process. The contiguous weight area refers to a dedicated high-speed storage area pre-allocated within the video memory of the first computing unit, which is logically or physically contiguous and used for compactly loading the target resident expert weights.

[0088] The specific methods for determining the target expert set that will preferentially reside in the first computing unit are as follows: A weighted frequency strategy is used, which introduces layer weights or position decay factors when calculating the original activation counts of each expert. This gives higher allocation priority to layers closer to the output layer or with greater computational overhead. For example, for two experts who have been activated 100 times, if one expert is located in a mixed expert layer closer to the output, or if its layer would result in higher computational latency when executed on the second computing unit, then the activation count of that expert can be multiplied by a larger layer weight. This results in a higher weighted activation score during sorting, and then experts are selected to join the target expert set according to the weighted activation scores from highest to lowest. Alternatively, a clustering strategy can be used: clustering analysis is performed on the activation pattern vectors of experts to identify expert groups with similar activation behaviors. Representative experts are selected from each expert group to join the target expert set to cover diverse activation patterns. Alternatively, a combinatorial optimization strategy can be used: the placement of experts is modeled as an integer linear programming or knapsack problem, with the optimization objective of maximizing the expected hit rate. The optimal allocation scheme is solved under the constraint of the global hardware resource budget to determine the target expert set.

[0089] In a distributed deployment scenario containing multiple first computing units: the computational load of the large model forward inference is distributed to multiple first computing units for parallel execution through tensor parallelism; the global expert scheduling state is uniformly generated by the main process node and synchronized to all other process nodes participating in tensor parallelism through set communication to ensure that all processes have a consistent view of the expert's residence location.

[0090] The global expert scheduling state can also be other data structures or representations, such as: an integer index list, which maintains a list or set of integers for each layer's GPU expert ID, implicitly expressing global budget constraints through list length and content; bitmap encoding, which compresses the expert placement information of each layer into a bitmap, storing mask information in a more compact way, reducing memory usage when the number of experts is extremely large; and a hierarchical configuration table, which uses an independent configuration method for each layer, specifying the number of GPU experts and the set of expert IDs for each layer, ensuring that the total does not exceed the global budget through external constraints. Furthermore, cross-engine mask sharing is not limited to fixed memory pointer passing; it can also be implemented using shared file mapping mmap, inter-process communication (IPC), or serialization synchronization mechanisms.

[0091] A method and system for optimizing heterogeneous hybrid expert large model inference based on dynamic expert scheduling prioritizes high-frequency active experts to reside on the GPU for execution, while non-resident experts are stored or computed on the CPU. Dual-stream parallelism reduces the impact of CPU-side data preparation, task submission, and result feedback on the main execution path, thereby improving GPU expert hit rate and end-to-end execution efficiency for MoE large model heterogeneous inference while maintaining model accuracy. With a hardware combination of consumer-grade GPUs and conventional CPU / DRAM, this method significantly improves the heterogeneous inference performance of MoE large models without increasing hardware costs or sacrificing model accuracy, effectively overcoming the bottlenecks of existing heterogeneous inference solutions in GPU hit rate, runtime adaptability, and CPU-GPU parallel efficiency.

[0092] like Figure 2 As shown, the state initialization unit, the concurrent execution unit, and the adaptive update unit contain multiple underlying sub-modules. The modules are connected and share states through a multidimensional Boolean tensor, which serves as the global scheduling state, i.e., the global GPU expert mask.

[0093] Specifically:

[0094] The state initialization unit includes, at its core, an expert activation statistics module, a first expert placement decision module, and a mapping module. The expert activation statistics module records the historical activation counts or frequencies of each expert based on the output of the gating network. The first expert placement decision module uses the data from the expert activation statistics module to generate the multidimensional Boolean tensor under global budget constraints. The mapping module loads the selected expert weights into a first computing unit (e.g., a GPU) based on the Boolean tensor and establishes an index mapping table from logical numbers to physical slots.

[0095] The concurrent execution unit comprises an execution module and a synchronization module at its underlying level. The execution module reads the shared multidimensional Boolean tensor and dispatches expert computation requests that hit the first computing unit (GPU) to the main execution path, while requests that miss are dispatched to the independent execution path of the second computing unit (CPU). The synchronization module is responsible for restoring and outputting the computation results from both sides in the routing order.

[0096] The adaptive update unit includes a runtime dynamic update module at its underlying layer. When preset triggering conditions are met, such as the number of tags in the model inference filling stage reaching a preset threshold, or the actual hit rate being lower than a preset threshold, the runtime dynamic update module recalculates the resident expert set and updates the multidimensional Boolean tensor and its copy on the first computing unit side, as well as the index mapping table in-situ.

[0097] The phrase "without interrupting the continuity of the main execution path computation" means that when dynamically replacing resident experts at runtime, it is not necessary to pause the inference pipeline of the first computation unit, nor is it necessary to rebuild the static computation graph used to accelerate decoding, thereby avoiding computational pauses or system lag. Specifically, in terms of timing control, the dynamic evaluation and copying of new expert weights are triggered and executed during the model filling phase, which does not rely on the static computation graph, completely isolating the update overhead from the latency-sensitive decoding phase. In terms of memory management, in-situ numerical overwriting is used, directly overwriting the newly selected target expert weights to the original physical slots in the continuous weight area, and simultaneously overwriting the global expert scheduling state and its copy. Throughout the entire overwriting and update process, the underlying physical memory address of all related tensors remains absolutely unchanged.

[0098] The following examples illustrate the heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling.

[0099] Example 1

[0100] Step S1: Global state initialization.

[0101] Read the relevant configuration, including at least the number of model layers, the number of experts per layer, and the global hardware resource budget. The global hardware resource budget is the global GPU expert budget, usually expressed as the maximum number of experts that can be accommodated, K.

[0102] The experts at all layers of the entire model are used as a unified candidate pool. First, the expert activation distribution data generated during the offline calibration phase is obtained. All candidate experts are globally sorted according to their activation frequency from high to low. The experts with the highest frequency are then added to the target expert set until the upper limit of the K value is reached.

[0103] Subsequently, a global expert scheduling state is generated for global overall routing. Specifically, the global expert scheduling state is instantiated as a multidimensional Boolean tensor with dimensions of [number of model layers, number of experts per layer]. The selected expert's corresponding position is set to True, and the unselected expert's position is set to False. Simultaneously, based on the Boolean tensor, an index mapping table is established from the logical expert number of the large model to the physical slot number of the continuous weight area within the first computing unit to solve the physical addressing problem of the weights.

[0104] Step S2: Heterogeneous dual-stream concurrent execution.

[0105] In forward inference of large models, especially during the decoding stage, the MoE model employs a routing network selection mechanism, Topk, which selects the k highest-scoring experts from all available experts to participate in the computation of the current token. k is typically 1 to 8, determining the actual computational cost and expert parallelism for each token. The routing network selection mechanism, Topk...

[0106] The multidimensional Boolean tensor is allocated in pinned memory, enabling consistent sharing of the view between the first inference engine responsible for scheduling the first computing unit (GPU) and the second inference engine responsible for scheduling the second computing unit (CPU).

[0107] During execution, on the main execution path of the first computation unit, the input tensor of the current layer is first copied to the staging buffer. The independent execution path of the second computation unit detects the completion of the copy via event synchronization, reads the immutable data copy from the buffer, and independently executes the expert inference on the CPU side of the second computation unit. Simultaneously, the main execution path executes the expert inference on the GPU side of the first computation unit in parallel on the original input tensor. After the main execution path completes its computation, it waits for the independent execution paths to complete via event synchronization and then merges the output results.

[0108] Step S3: Dynamic adaptive update in place at runtime.

[0109] To address unpredictable workload changes during inference service operation, actual activation features are extracted during operation to trigger dynamic evaluation and redetermine the target expert set for the first computing unit.

[0110] During the update phase, the newly selected expert weights are copied to the continuous weight region of the first computation unit, and the global expert scheduling state, i.e., the multidimensional Boolean tensor, its copy on the first computation unit side, and the index mapping table are overwritten in place, while keeping the memory addresses of the multidimensional Boolean tensor and the index mapping table unchanged, thereby ensuring compatibility with the captured static computation graph CUDA Graph and avoiding high reconstruction overhead.

[0111] Example 2

[0112] Workload OOD (Out-of-Distribution) refers to a scenario where there is a difference between the actual user request and the offline calibration data in the expert activation mode, resulting in a sharp drop in the first computing unit GPU expert hit rate, i.e., the proportion of activated experts located on the first computing unit GPU. The triggering conditions for extracting the actual activation features of the current workload as a feedback signal to trigger dynamic evaluation include any one or more of the following combinations.

[0113] Based on the number of tokens: During the prefill phase of large model inference, the number of tokens in the current batch reaches a first preset threshold. The first preset threshold is a preset value.

[0114] Based on a time period: Since the last update of the global expert scheduling state, the continuous running time of the inference service reaches a second preset period. The second preset period is a preset value. During operation, the time of completion of the last dynamic update is recorded. When the interval between the current time and the last update time reaches the second preset period, the dynamic update process is started in the prefill phase of the next executable update. This method does not rely on the number of tokens in a single request, nor on instantaneous hit rate judgment, and is suitable for scenarios where the request load is relatively stable but may change slowly over time.

[0115] Based on performance feedback: Within a preset operating window, the actual hit rate of the target expert set residing in the first computing unit is lower than a third preset threshold. The third preset threshold is a pre-defined value. Specifically, during inference, the proportion of experts actually activated that are matched with GPU-residing experts is statistically analyzed to obtain the GPU expert hit rate for the current batch, current time window, or cumulative interval. When this hit rate is lower than the third preset threshold, it indicates that the current expert placement result does not match the actual workload, triggering a dynamic update in the prefill stage of the next executable update. In practice, the number of GPU expert hits and the total number of activated experts can be recorded separately, and their ratio can be calculated. When several consecutive statistical results are lower than the third preset threshold, an expert mask update is performed to avoid frequent updates caused by occasional requests.

[0116] Based on request accumulation: Since the last update of the global expert scheduling state, the cumulative number of inference requests processed has reached a fourth preset threshold. The fourth preset threshold is a preset value. Specifically, it maintains the number of requests or batches processed since the last dynamic update. When the cumulative value reaches the fourth preset threshold, the next expert mask update is triggered, and the counter is reset after the update is complete. In practice, the number of requests can be counted per individual user request or per batch processing time; in continuous service scenarios, it can also be combined with the minimum update interval to avoid multiple updates triggered in a short period of time due to high concurrency. Request accumulation-based updates are suitable for scenarios with a relatively stable request scale where it is desirable to periodically refresh the expert placement results according to the service processing volume.

[0117] When dynamic evaluation is triggered during the prefill phase of large model inference, due to the lack of prior knowledge of activations in the current workload's OOD text, a fallback mode is constructed on the first computational unit to complete the forward propagation of the current batch. This process, regardless of cost, pulls all experts from this layer into the first computational unit to fully obtain the actual expert activation distribution of the current batch in the output of the gating network, serving as the basis for redetermining the target expert set for the subsequent decoding phase.

[0118] The Prefill phase refers to the process where, before the generation task begins, the model performs a complete forward computation on the user-provided context text prompt. Its core functions are: calculating a self-attention key-value pair cache: For each existing token in the context text prompt, the Transformer calculates the corresponding key and value at each layer using a self-attention module and stores them in the cache; generating contextual representations: Simultaneously, the model outputs the hidden states at each layer, providing rich contextual information for subsequent generation stages; and completing the most costly part in one go: Since the prompt is often long, Prefill requires performing complete attention computation at each position, making it the most time-consuming step in the inference process. The processing speed of the Prefill phase affects the time to generate the first token, i.e., Time To First Token (TTFT).

[0119] The Decode phase refers to the process after prefilling, where the model gradually generates new tokens. Each time a new token is generated, the model only needs to perform attention calculations between the query at that position and the previously cached key-value pairs, without repeatedly calculating the entire context. This reduces the computational cost per step from O(n²) to O(n), where n is the number of tokens generated. The newly generated token is converted into its corresponding key-value pair and appended to the cache for use in the next decoding step; simultaneously, the hidden state of the last layer is updated to keep up with context changes.

[0120] Example 3

[0121] In addition to absolute frequency ranking, the methods for determining the target expert set may also include any of the following.

[0122] Weighted Frequency Strategy: When calculating the original activation counts of each expert, layer weights or position decay factors are introduced to give higher allocation priority to experts in layers closer to the output layer or with higher computational overhead. Their original activation counts are multiplied by the weight factor to obtain a weighted activation score, which is then used for global descending selection. Specifically, the original activation counts c(l,e) of the e-th expert in layer l are first recorded, and then a layer weight a(l) is set for each layer. Layer weights can be pre-set based on layer position or historical GPU hit rate, or generated after statistics are collected during the calibration phase. Subsequently, the weighted score s(l,e) = a(l) × c(l,e) is calculated for each expert, and all experts in the model are sorted according to s(l,e). The expert with the highest score within the global GPU expert budget is selected and written into the global mask.

[0123] Clustering strategy: Cluster analysis is performed on the activation pattern vectors of each expert to identify expert groups with similar activation behaviors. Representative experts are then extracted from each expert group and added to the target expert set to enhance the coverage of diverse activation patterns.

[0124] Combinatorial optimization strategy: The placement of experts is abstracted into an integer linear programming (ILP) problem or a classic knapsack problem. The global hardware resource budget is used as the knapsack capacity constraint, and the value optimization objective is to maximize the expected hit rate. The theoretically optimal expert allocation scheme is then solved.

[0125] The activation time-weighted strategy retains the time information of activation events when counting the number of expert activations, and sets attenuation weights according to the time proximity. Specifically, it records the timestamp or request number of each expert activation event, calculates the time distance between each activation event and the current time, and calculates the corresponding attenuation weight according to a preset attenuation function. The larger the time distance, the smaller the attenuation weight. The preset attenuation function includes an exponential attenuation function or a piecewise attenuation function. The attenuation weights of all activation events of the same expert are weighted and summed to obtain the time-attenuated activation score. Experts are selected to be added to the target expert set according to the activation scores from highest to lowest. In specific implementation, a timestamp or request number is recorded for each expert activation event. For an activation event occurring before the current time T, its time distance is denoted as Δt. The weight w(Δt) of the event is calculated according to the preset attenuation function, where the larger Δt is, the smaller the weight. For example, an exponential attenuation function w(Δt) = exp(-λΔt) or a piecewise attenuation function can be used. Subsequently, a weighted summation is performed on all activation events of the same expert to obtain the activation score after time decay. Experts in the entire model are then ranked according to this score, and the expert with the highest score within the global GPU expert budget is selected and written into the global mask.

[0126] Sliding time window statistical strategy: This strategy involves counting the number of expert activations within a recent time window. A preset-length time window is maintained. When a new inference request or batch completes routing, the activation count of each expert is added to the statistical value of the time window. When a historical inference request or batch exceeds the time window's range, the corresponding activation count is deducted from the statistical value. Experts are then selected from the cumulative activation counts within the current time window, ranked from highest to lowest, and added to the target expert set. Specifically, a time window of length W is maintained. When a new request or batch completes routing, the activation count of each expert is added to the window statistics. When an old request or batch exceeds the window's range, the corresponding activation count is deducted from the statistical results. All model experts are sorted based on their cumulative activation counts within the current window, and the expert with the highest activation count within the global GPU expert budget is written into the global mask.

[0127] Example 4

[0128] In a distributed deployment scenario that includes multiple of the first computing units, tensor parallelism and FP8 quantization format compression are employed.

[0129] The computational load of the large model forward inference is distributed across multiple first computing units for parallel execution using tensor parallelism. The global expert scheduling state is uniformly generated by the master process node and synchronized to all other process nodes participating in tensor parallelism via aggregate communication to ensure that all processes maintain a consistent view of the expert's location. Tensor Parallelism (TP) refers to a distributed deployment scenario containing multiple first computing units, such as 4-card or 8-card nodes, where the model forward inference computation is distributed along a specific dimension to multiple computing devices for parallel execution using a distributed computing strategy (TP), and then the results are merged via aggregate communication, such as AllReduce. In this scenario, the global expert scheduling state is uniformly generated by the master process node Rank 0 and then synchronized to all other process nodes participating in tensor parallelism via aggregate communication to ensure absolute consistency of the expert's location view.

[0130] FP8 quantization compression is designed to better suit the capacity limitations of consumer-grade graphics cards. The weights of experts in large models, including gated projection weights, up-projection weights, and down-projection weights, are quantized using 8-bit floating-point FP8. Compared to conventional FP16 or FP32, FP8 halves storage requirements with minimal impact on model accuracy and significantly accelerates matrix multiplication operations on the main execution path of the first computation unit. Combined with a dynamic scheduling scheme, this can achieve a significant increase in throughput.

[0131] The heterogeneous hybrid expert large model inference optimization system based on dynamic expert scheduling has its underlying physical architecture deployed in an electronic device or server cluster containing heterogeneous computing resources. Specifically, this underlying hardware architecture includes: at least one central processing unit (CPU) serving as the second computing unit, a large-capacity DRAM connected to the CPU, and at least one graphics processing unit (GPU) serving as the first computing unit, and high-bandwidth video memory (HBM or GDDR) packaged together with the GPU. The CPU and GPU communicate with each other via a high-speed serial interconnect bus, such as a PCIe bus, for data and control commands.

[0132] The virtual functional modules such as the state initialization unit, concurrent execution unit, and adaptive update unit described in this embodiment of the invention are essentially computer programs or instruction sets stored in the main memory or related non-volatile storage media and retrieved and executed by the CPU and / or GPU. Through this specific heterogeneous hardware architecture, this invention leverages the CPU's memory capacity advantage to store less popular expert data and utilizes the GPU's computing power and video memory bandwidth advantage to accelerate the inference calculations of popular expert data.

[0133] A heterogeneous hybrid expert model inference optimization method and system based on expert scheduling significantly improves the problems of low GPU expert hit rate and low heterogeneous execution efficiency through the synergy of global expert masking, activation frequency-based placement strategy, runtime dynamic expert updates, and CPU-GPU dual-stream parallel execution. Under the same GPU memory budget, it increases the actual amount of expert computation performed by the GPU by several times. Furthermore, when facing distributed off-target (OOD) workloads, it can adaptively adjust expert placement based on the actual activation mode at runtime. The dual-stream parallel execution method can restore throughput to nearly 100%, eliminating the anomaly that using a heterogeneous framework is slower than using a pure GPU. In summary, this invention, with a hardware combination of consumer-grade GPU and conventional CPU / DRAM, can significantly improve the heterogeneous inference performance of large MoE models without increasing hardware costs or sacrificing model accuracy, effectively breaking through the bottlenecks of existing heterogeneous inference schemes in GPU hit rate, runtime adaptability, and CPU-GPU parallel efficiency.

[0134] The above description illustrates preferred embodiments of the present invention and helps those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative and should not be construed as limiting the specific implementation of the present invention to these embodiments. For those skilled in the art, several simple deductions and modifications can be made without departing from the inventive concept, and all such modifications should be considered within the protection scope of the present invention.

Claims

1. A heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling, characterized in that, Includes the following steps: Step S1: Based on the set global hardware resource budget and the historical activation characteristics of each expert in the whole model, determine the target expert set that will preferentially reside in the first computing unit in the global scope, and generate a global expert scheduling state for global overall planning of expert computing task routing accordingly. Step S2: When performing forward inference of the model, according to the indication of the currently effective global expert scheduling state, the expert calculation tasks that have been successfully assigned to the main execution path of the first computing unit are dispatched, while the expert calculation tasks that have not been successfully assigned to the independent execution path of the second computing unit are dispatched; wherein, the main execution path and the independent execution path are processed asynchronously and concurrently in parallel, and the results are merged and output after both have been calculated; Step S3: During the execution of step S2, the actual activation features of the inference request currently being processed by the large model are extracted as a feedback signal to trigger dynamic evaluation, so as to redetermine the target expert set of the first computing unit; and without interrupting the computational continuity of the main execution path, the global expert scheduling state on which step S2 depends and the corresponding expert weights on the first computing unit are updated in place, so that subsequent forward inference tasks can continue to be executed asynchronously and concurrently based on the updated global expert scheduling state.

2. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The global expert scheduling state is specifically a multidimensional Boolean tensor, the dimension of which is determined by the number of layers in the large model and the number of experts in each layer; Step S1 further includes: based on the global expert scheduling state, establishing an index mapping table from the logical expert number of the large model to the physical slot number of the continuous weight area within the first computing unit.

3. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The determination of the target expert set that will preferentially reside in the first computing unit specifically includes: Obtain the expert activation distribution data generated during the offline calibration phase; Within the entire model, all experts are globally sorted from high to low according to the activation frequency in the activation distribution data. The experts with the highest activation frequency are selected in sequence and added to the target expert set until the upper limit of the global hardware resource budget is reached.

4. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The first computing unit is a graphics processing unit (GPU) containing high-bandwidth video memory, and the second computing unit is a central processing unit (CPU) that works in conjunction with main memory. In step S2, the global expert scheduling state is allocated in fixed memory and is shared consistently across computing devices between the first inference engine responsible for scheduling the first computing unit and the second inference engine responsible for scheduling the second computing unit.

5. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The asynchronous concurrent parallel processing of the main execution path and the independent execution path specifically includes: On the main execution path of the first computing unit, the input tensor of the current layer is copied to the temporary buffer; The independent execution path of the second computing unit detects the completion of the copy through event synchronization, reads the immutable data copy in the temporary buffer, and independently executes the expert inference calculation of the second computing unit; while the main execution path executes the expert inference calculation of the first computing unit in parallel on the original input tensor. After the main execution path completes its calculation, it waits for the independent execution path to complete its calculation via event synchronization in order to merge the two calculation results.

6. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The triggering conditions for extracting the actual activation features currently being processed by the large model as feedback signals to trigger dynamic evaluation include: During the population phase of model inference, the number of tags in the current batch reaches a first preset threshold; or, Since the last update of the global expert scheduling status, the continuous runtime of the inference service has reached the second preset period; or, Within a preset operating window, the actual hit rate of the target expert set residing in the first computing unit is lower than a third preset threshold; or, Since the last update of the global expert scheduling status, the cumulative number of inference requests processed has reached the fourth preset threshold.

7. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 6, characterized in that, When dynamic evaluation is triggered during the filling phase of the model inference, the forward propagation of the current batch is completed by constructing a fallback mode of a temporary full expert layer on the first computing unit, so as to fully obtain the actual expert activation distribution of the current batch in the gating network output, and use it as the basis for redetermining the target expert set.

8. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, 6, or 7, characterized in that, The global expert scheduling status and the corresponding expert weights on the first computing unit are updated in-situ. The specific methods include: Copy the weight data corresponding to the newly selected target expert set to the continuous weight area of ​​the first calculation unit; The global expert scheduling state, its copy on the first computing unit side, and the index mapping table are overwritten in place. The in-situ update process maintains the global expert scheduling state and its copy, as well as the memory address of the index mapping table, unchanged.

9. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, The specific methods for determining the target expert set that will preferentially reside in the first computing unit include: When calculating the original activation counts of each expert, layer weights or position decay factors are introduced to give higher allocation priority to layers closer to the output layer or with higher computational overhead. After obtaining the weighted activation scores, experts are selected from high to low according to the weighted activation scores to join the target expert set; or, Record the timestamp or request number of each expert activation event, calculate the time distance between each activation event and the current time, and calculate the corresponding decay weight according to a preset decay function, wherein the larger the time distance, the smaller the decay weight. The preset decay function includes an exponential decay function or a piecewise decay function. The decay weights of all activation events for the same expert are weighted and summed to obtain the time-decayed activation score. Experts are then selected from high to low activation scores to join the target expert set; or... Maintain a time window of a preset length. When a new inference request or batch is routed, the activation count of each expert is added to the statistical value of the time window. When a historical inference request or batch exceeds the range of the time window, the corresponding activation count is deducted from the statistical value. Experts are selected from high to low based on the cumulative activation count of each expert within the current time window and added to the target expert set.

10. The heterogeneous hybrid expert large model inference optimization method based on dynamic expert scheduling according to claim 1, characterized in that, In a distributed deployment scenario that includes multiple first computing units, the computational load of the large model forward inference is distributed to multiple first computing units in parallel through tensor parallelism for execution. The global expert scheduling state is generated uniformly by the master process node and synchronized to all other process nodes participating in tensor parallelism through set communication, so as to ensure that all processes have a consistent view of the expert's residence location.

11. A heterogeneous hybrid expert large model inference optimization system based on dynamic expert scheduling, characterized in that, include: The state initialization unit is used to determine the set of target experts who should reside in the first computing unit in the global scope based on the set global hardware resource budget and the historical activation characteristics of each expert in the whole model, and generate a global expert scheduling state for global overall planning of expert computing task routing. The concurrent execution unit is used to dispatch expert computation tasks that have been successfully executed to the main execution path of the first computing unit, and dispatch expert computation tasks that have not been successfully executed to the independent execution path of the second computing unit, according to the indication of the currently effective global expert scheduling state, during the forward inference of the execution model; wherein the main execution path and the independent execution path are processed in parallel through an asynchronous concurrency mechanism, and the results are merged and output after both have been computed; An adaptive update unit is used to extract the actual activation features of the inference request currently being processed by the large model as a feedback signal to trigger dynamic evaluation during the operation of the concurrent execution unit, so as to redetermine the target expert set of the first computing unit; and without interrupting the computational continuity of the main execution path, update the global expert scheduling state on which the concurrent execution unit depends and the corresponding expert weights on the first computing unit in place, so that subsequent forward inference tasks can continue to be executed asynchronously and concurrently based on the updated global expert scheduling state.

Images