A multi-card expert balanced reasoning method and system for a hybrid expert model

Abstract

The application provides a multi-card expert balanced reasoning method and system for a mixed expert model, and is applied to a mixed expert model reasoning stage. The method obtains an expert deployment matrix and an expert load vector, determines a set of mappable devices of each expert, and divides the experts into rigid experts and elastic experts; according to the unique deployment position of the rigid experts, the method completes deterministic allocation and initializes device load information; for the elastic experts, the method performs batch cascading routing based on the expert deployment matrix and the current device load information, determines target computing devices, and dynamically updates the device load information; according to the routing results of the rigid experts and the target computing device indexes of the elastic experts, the method constructs a complete expert routing matrix, and generates an expert token load vector corresponding to the current computing device. Through differentiated expert scheduling, device-level load balancing in the multi-card reasoning process is realized, and the utilization rate of computing resources and the reasoning efficiency of the mixed expert model are improved.

Description

Technical Field

[0001] This invention belongs to the field of hybrid expert models, specifically relating to a multi-card expert equilibrium inference method and system for hybrid expert models. Background Technology

[0002] With the rapid development of large-scale language models, Mixture of Experts (MoE) models have been widely applied in artificial intelligence fields such as natural language processing, intelligent question answering, and code generation due to their advantage of significantly improving model parameter scale and inference performance while maintaining low computational cost. In MoE models, tokens are the smallest input units that the model can process. Each input token activates only a small number of experts to participate in the computation. Experts are dynamically selected through a gate network, thereby achieving sparse computation and significantly reducing the overall computational overhead. However, due to the significant differences in the activation probabilities of different experts, expert access hotspots are prone to occur during distributed inference, leading to uneven load distribution among different computing nodes and affecting the overall inference efficiency of the system. Therefore, how to achieve efficient expert scheduling and load balancing during the inference process of MoE models has become one of the important research directions of current distributed inference systems.

[0003] To address the aforementioned issues, existing technologies have proposed various expert load balancing optimization methods for the MoE inference phase, among which the Expert Parallelism LoadBalancing (EPLB) mechanism proposed by DeepSeek is particularly representative. This type of method is primarily designed for high-concurrency prefill phases or large-scale inference scenarios. By splitting and remapping input tokens and deploying redundant experts, tokens that would otherwise be concentrated on a few hot experts can be distributed across multiple expert replicas for execution. This reduces the access pressure on hot experts, achieves statistical load balancing, and improves overall computing resource utilization.

[0004] However, existing EPLB schemes primarily rely on the premise of a large number of tokens being input in parallel during the Prefill phase, with their optimization objective focused on token-level computational load balancing. In actual deployment, most of the inference time of the MoE model is typically consumed in the Decoding inference phase. Because the Decoding phase uses an autoregressive generation method, generating only a very small number of tokens at each step, the entire inference process exhibits low concurrency and serial execution characteristics, fundamentally different from the Prefill phase. At this stage, the prerequisite of achieving load balancing based on a large number of token splits and redundant mappings no longer exists, making it difficult for existing EPLB mechanisms to continue functioning effectively.

[0005] Furthermore, the system performance bottleneck in the Decoding stage has also changed significantly. Compared to the computationally intensive nature of the Prefill stage, which is mainly based on matrix calculations, the number of input tokens in the Decoding stage is reduced from batch sequences to single-step autoregressive generation. The computational cost of matrix multiplication within each expert network (proportional to the number of tokens) has decreased significantly. However, the HBM transport of expert parameters (related to the number of model parameters) remains almost unchanged, and the arithmetic intensity (computational cost / memory access) in a single forward computation decreases sharply. This leads to the memory access and communication overhead caused by loading, transporting, and updating expert parameters between different computing cards becoming the main factor affecting inference latency. In other words, the system bottleneck gradually shifts from computationally intensive to memory bandwidth-constrained. Under these operating characteristics, the overall system performance depends more on the balance of expert weight loading times and memory access pressure among the computing cards, rather than solely on the token distribution ratio among experts.

[0006] However, existing EPLB methods still use token-level load balancing as the optimization objective, without establishing effective balancing constraints for the total number of expert activations actually handled by each compute card during the Decoding phase. When hot experts are continuously concentrated on a few compute cards, those cards need to frequently load and move expert weights, while other compute cards remain relatively idle. This leads to a continuous increase in inter-card memory access pressure and I / O load differences, significantly impacting the overall system throughput and inference stability. Since there is no direct correlation between token distribution balance and the frequency of expert weight movement, existing solutions cannot fundamentally reduce the imbalance in cross-card weight access during the Decoding phase, and their effect on improving end-to-end inference latency is limited.

[0007] In summary, existing EPLB-based load balancing methods are mainly suitable for the high-concurrency Prefill stage. For the low-concurrency Decoding stage, they suffer from insufficient applicability, biased understanding of system bottlenecks, and mismatched optimization goals. They struggle to ensure consistency in the number of expert activations and weight transfer load across computing cards, thus failing to effectively reduce cross-card memory access overhead and improve the overall performance of the distributed MoE inference system. Therefore, there is an urgent need to propose a load balancing method suitable for the MoE model Decoding stage. This method should, while ensuring the correctness of model inference, aim to balance the total number of expert activations across computing cards, thereby optimizing expert scheduling, reducing weight transfer overhead, and improving inference throughput and system stability. Summary of the Invention

[0008] This invention provides a multi-card expert balanced inference method and system for hybrid expert models, in order to solve the problems in the prior art where it is difficult to achieve device-level load balancing during the expert scheduling process in the inference phase of hybrid expert models, resulting in concentrated access by hot experts, low utilization of cross-device computing resources, and high inference latency.

[0009] To address the aforementioned technical problems, the present invention discloses the following technical solutions: One aspect of the present invention provides a multi-card expert equilibrium inference method for hybrid expert models, applied in the inference stage, comprising: Obtain the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert; Experts with a set of mappable devices of size 1 are defined as rigid experts, and experts with a set of mappable devices of size greater than 1 are defined as elastic experts. Based on the unique deployment location of the rigid experts, the active rigid experts are assigned to the corresponding computing devices, and the device load information is initialized based on the number of active rigid experts handled by each computing device. For the elastic experts in the current inference batch, batch cascading routing is performed based on the expert deployment matrix and the current device load information to determine the target computing device corresponding to each elastic expert, and the device load information is updated after each processing batch is completed. Construct a complete expert routing matrix based on the rigid expert routing results and the elastic expert target calculation device index; Generate the expert term load vector corresponding to the current computing device based on the complete expert routing matrix.

[0010] Optionally, obtaining the expert deployment matrix and expert load vector, and determining the set of mappable devices for each expert, includes: Obtain an expert deployment matrix to characterize the deployment relationships of experts on different computing devices; Obtain the expert load vector, which represents the load distribution information of each expert during the reasoning process; The set of mappable devices corresponding to each expert is determined based on the expert deployment matrix.

[0011] Optionally, obtaining the expert load vector, used to characterize the load distribution information of each expert during the inference process, includes: Obtain the Top-k expert gating results output by the hybrid expert model; The number of tokens assigned to each expert is calculated based on the Top-k expert gating results. The number of terms corresponding to each expert is used to construct the expert load vector.

[0012] Optionally, the method further includes: Convert the data format of the expert deployment matrix and expert load vector to INT32 type.

[0013] Optionally, the step of assigning active rigid experts to corresponding computing devices based on their unique deployment locations, and initializing device load information based on the number of active rigid experts handled by each computing device, includes: The number of deployment replicas for each expert is calculated based on the expert deployment matrix, and the active rigid experts in the current inference batch are determined by combining the expert load vector. The expert deployment matrix is ​​filtered based on the rigid expert mask to generate a rigid expert routing matrix; The rigid expert routing matrix is ​​summed along the expert dimension to obtain the number of active rigid experts supported by each computing device, which is then used as the initial load information for the device.

[0014] Optionally, the step of performing batch cascading routing for elastic experts in the current inference batch based on the expert deployment matrix and current device load information, determining the target computing device corresponding to each elastic expert, and updating the device load information after each processing batch is completed includes: Based on the expert deployment matrix, extract the deployment relationships corresponding to active elastic experts and generate an elastic expert deployment submatrix; Static reachability biases are constructed based on the elasticity expert deployment submatrix. Active elastic experts are divided into multiple processing batches arranged at intervals according to the preset number of processing steps; For each elastic expert in the current processing batch, obtain the score value of each computing device based on the static reachability bias and the current device load information; Select the computing device with the highest score as the target computing device for the current elasticity expert; Update the device load information of each computing device based on the elastic expert allocation results in the current processing batch.

[0015] Optionally, constructing the complete expert routing matrix based on the rigid expert routing results and the resilient expert target computing device index includes: Initialize the complete expert routing matrix; Write the routing results of the active rigid experts into the complete expert routing matrix; Write the target device index of the active elastic expert into the complete expert routing matrix.

[0016] Optionally, generating the expert lexical load vector corresponding to the current computing device based on the complete expert routing matrix includes: Extract the expert routing mask corresponding to the current computing device; Generate the current device term load vector arranged according to the global expert ID; Generate local expert terminology payload vectors based on the expert deployment relationships of the current computing devices; Output the expert lexical load results corresponding to the current computing device.

[0017] Another aspect of the present invention provides a multi-card expert equilibrium inference system for hybrid expert models, applied in the inference phase, comprising: The input processing module is configured to acquire the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert; The expert classification module is configured to identify experts whose set of mappable devices is 1 as rigid experts and experts whose set of mappable devices is greater than 1 as flexible experts. The rigid expert allocation module is configured to allocate active rigid experts to corresponding computing devices based on the unique deployment location of the rigid experts, and initialize device load information based on the number of active rigid experts hosted by each computing device. The Elastic Expert Allocation Module is configured to perform batch cascading routing for the elastic experts in the current inference batch based on the expert deployment matrix and the current device load information, determine the target computing device corresponding to each elastic expert, and update the device load information after each processing batch is completed. The expert routing matrix construction module is configured to construct a complete expert routing matrix based on rigid expert routing results and elastic expert target calculation device indexes. The expert terminology load vector generation module is configured to generate the expert terminology load vector corresponding to the current computing device based on the complete expert routing matrix.

[0018] This invention provides a multi-GPU expert balanced inference method and system for hybrid expert models. During the inference phase, experts are first categorized into rigid and flexible experts based on their deployment relationships. Rigid experts are fixedly deployed using a deterministic device mapping method, while flexible experts are dynamically scheduled based on current device load information. By employing differentiated scheduling strategies for different types of experts, the problem of hot experts concentrating on a small number of computing devices is effectively avoided while ensuring expert deployment constraints. This improves the utilization rate of multi-GPU computing resources and reduces computational waiting and resource idleness caused by uneven device load during the inference phase.

[0019] Furthermore, after completing expert scheduling, this invention constructs a complete expert routing matrix and generates corresponding expert term load vectors, enabling the expert scheduling results and device load status to form a unified structured representation, providing standardized routing information for subsequent hybrid expert model inference computation. This invention achieves dynamic load balancing without complex global optimization solutions, offering advantages such as simple implementation, high scheduling efficiency, and ease of engineering deployment. It can effectively improve the overall inference efficiency and system throughput performance in multi-GPU hybrid expert model inference scenarios.

[0020] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify essential or necessary features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0022] Figure 1 A flowchart illustrating a multi-card expert equilibrium inference method for hybrid expert models provided in an embodiment of the present invention; Figure 2 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S100; Figure 3 An implementation provided by an embodiment of the present invention Figure 2 A flowchart illustrating step S102; Figure 4 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S300; Figure 5 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S400; Figure 6 An implementation provided by an embodiment of the present invention Figure 1 A flowchart of step S500; Figure 7 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S600; Figure 8 This is a schematic diagram of the structure of a multi-card expert balanced inference system for hybrid expert models provided in an embodiment of the present invention. Detailed Implementation

[0023] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0024] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] Figure 1 This is a flowchart illustrating a multi-card expert equilibrium inference method for hybrid expert models provided in an embodiment of the present invention. This method is applied to the inference stage of a hybrid expert model, such as... Figure 1 As shown, it includes the following steps: Step S100: Obtain the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert.

[0026] In one embodiment of the present invention, such as Figure 2 As shown, step S100 can be implemented in the following manner: Step S101: Obtain the expert deployment matrix to characterize the deployment relationship of each expert on different computing devices.

[0027] During the model deployment phase, the system allocates each expert instance to different computing devices to perform loading and binding operations based on a preset expert parallel strategy or a manually configured deployment scheme. After completing the mapping configuration from experts to computing devices, the system expresses this mapping relationship in a structured matrix form, generating the expert deployment matrix P.

[0028] The expert deployment matrix P represents the deployment relationship of experts on different computing devices. It is a two-dimensional structured matrix, with rows corresponding to different computing devices (including but not limited to GPU cards, NPU cards, or distributed computing nodes) and columns corresponding to different expert instances. The matrix element P(g,e) represents the deployment status of expert e on computing device g. When P(g,e)=1, it means that the expert has been deployed on the corresponding device, and when P(g,e)=0, it means that the expert has not been deployed on the device, thus forming a topological mapping relationship of experts among multiple devices.

[0029] Step S102: Obtain the expert load vector, which is used to characterize the load distribution information of each expert during the reasoning process.

[0030] In one embodiment of the present invention, such as Figure 3 As shown, the expert load vector can be obtained in the following ways: Step S1021: Obtain the Top-k expert gating results output by the hybrid expert model.

[0031] During the inference process of the MoE model, the gating network calculates the expert selection result for each input token and selects the top-k experts with the highest probability values ​​from all experts as the activation expert set for that token, thus generating the Top-k expert gating result. The Top-k expert gating result is typically represented in index form, recording the k expert numbers assigned to each token, thus forming a sparse mapping relationship between tokens and experts. In one implementation, this result can also be represented as a two-dimensional index structure, where the first dimension corresponds to the token sequence number, and the second dimension corresponds to the Top-k expert numbers selected for that token.

[0032] Step S1022: Calculate the number of tokens assigned to each expert based on the Top-k expert gating results.

[0033] The Top-k expert indexes for all tokens are expanded, and the number of times each expert is selected in the current inference batch is accumulated, using expert ID as the statistical dimension. One implementation method is to use histogram statistics or bincounting to achieve this process. This involves iterating through the Top-k expert list corresponding to all tokens, treating each expert selection as a count accumulation, thus obtaining the total number of tokens allocated to each expert. This statistical result not only reflects the activation frequency of experts in the current inference input but also corresponds to their actual computational load intensity, effectively characterizing the load differences between different experts.

[0034] Step S1023: Construct the expert load vector by calculating the number of lexical units corresponding to each expert.

[0035] After completing the above statistics, the token counts for each expert are arranged according to the expert index order, forming a one-dimensional vector structure, namely the expert load vector T. The e-th component T in the vector... e This represents the number of tokens received or the number of times expert e is activated in the current inference batch. This expert load vector is used to uniformly characterize the load distribution of all experts in the current inference phase, providing key input for subsequent classification of rigid and elastic experts, device-level load updates, and load-penalty-based scheduling optimization.

[0036] Step S103: Determine the set of mappable devices corresponding to each expert based on the expert deployment matrix.

[0037] For any expert e, iterate through its column vector P(:,e) in the expert deployment matrix P, and sum the elements in the column vector to obtain the number of times the expert is deployed across all computing devices, denoted as the number of replicas R. e .

[0038] When R is calculated e When R = 1, it means that the expert is deployed on a single computing device, and in this case, the expert has a unique reachable device constraint during the scheduling process; when R = 1, it means that the expert is deployed on a single computing device, and in this case, the expert has a unique reachable device constraint. e When the value is greater than 1, it indicates that the expert has redundant copies deployed on multiple computing devices, and the expert has multiple device selection options during the scheduling process.

[0039] The system traverses and analyzes the deployment relationships of experts across all computing devices, and determines the reachable device range based on the position of the non-zero element in the corresponding expert column vector in the deployment matrix. Since the expert deployment matrix P already describes the mapping relationship between experts and computing devices in a structured form, the deployment status of any expert e on different computing devices can be directly reflected by the matrix P(:,e).

[0040] In one optional implementation, for each expert e, the system sequentially scans its column vector data in the expert deployment matrix P, and filters and collects all computing devices g that satisfy P(g,e)=1, where g represents the computing device index. The set of all computing devices that meet the deployment conditions together constitutes the set of mappable devices for that expert. ,Right now:

[0041] For an expert with one replica, the set of mappable devices degenerates into a single-element set, thus corresponding to a uniquely determined computing device; for an expert with more than one replica, the set of mappable devices contains multiple candidate computing devices.

[0042] In one embodiment of the present invention, it is also necessary to perform data type conversion on the expert deployment matrix and the expert load vector, and unify the data representation format of the two to the INT32 type.

[0043] In the MoE expert scheduling system, to improve fault tolerance and load balancing capabilities, additional elastic experts (i.e., multiple replicas of the same expert deployed on multiple computing devices) are introduced. The expert deployment matrix is ​​generated based on the real-time distribution of these elastic and rigid experts, representing the deployment relationship of experts on different computing devices. The expert load vector comes from the Top-k gating statistics of the routing module. Its raw data may be represented in INT64, FLOAT, or mixed precision form, reflecting the number of tokens allocated to each expert or the number of activations during the inference process. Due to the differences in the generation source and storage format of the above two types of data, if they are directly used in subsequent matrix summation, index mapping, and load update calculations, it may lead to data overflow, implicit type conversion, or inconsistent calculation precision.

[0044] Therefore, in this embodiment, after the system obtains the expert deployment matrix and expert load vector, it performs a unified data type conversion process on them. Specifically, each element in the expert deployment matrix is ​​converted from its original Boolean type or low-order integer type to an INT32 integer type, where a value of 1 indicates that there is a deployment relationship between the expert and the computing device, and a value of 0 indicates that there is no deployment relationship. This ensures that the matrix has a unified integer calculation semantics when participating in subsequent summation, statistical and index calculations.

[0045] Simultaneously, each component in the expert load vector undergoes a unified conversion, transforming it from its original high-precision integer or floating-point type to an INT32 type. This conversion serves to uniformly represent the token allocation quantity or activation count for each expert in the current inference batch. In one implementation, if the original load value is a floating-point number, it is converted to an integer form through rounding or truncation to ensure consistent integer semantics with subsequent device load accumulation operations. If the original load value is a high-order integer, it is mapped to the INT32 representation range through type truncation or secure type casting.

[0046] Step S200: Determine experts whose set of mappable devices has a size of 1 as rigid experts, and determine experts whose set of mappable devices has a size greater than 1 as elastic experts.

[0047] After constructing the set of mappable devices for each expert, the system further classifies all experts based on the size of their corresponding set of mappable devices, thereby establishing scheduling strategies for different experts. Since experts have different deployment methods in multi-GPU environments, the scheduling degrees of freedom they possess during inference also differ. This invention classifies experts, allowing experts with unique deployment constraints and those with multi-replica deployment capabilities to use different scheduling methods, thus decoupling deterministic scheduling from dynamic load balancing scheduling.

[0048] In one implementation, the system first calculates the number of devices in the set of mappable devices A(e) corresponding to each expert, i.e., the size of the set. .in, This represents the set of mappable devices corresponding to expert e. This indicates the number of computing devices in the set. When When =1, it indicates that the expert is deployed on only one computing device, and there is only one executable device available for it throughout the entire inference process. It has no device selection space, therefore this expert is classified as a rigid expert; when When the value is greater than 1, it indicates that the expert is deployed on multiple computing devices simultaneously and can choose between multiple devices during the inference process. Therefore, the expert is identified as a flexible expert.

[0049] For rigid experts, since their corresponding set of mappable devices contains only a single computing device, their scheduling result is entirely determined by the deployment relationship. There is no need for device search, scoring calculation, or load balancing optimization. Their subsequent scheduling process is completed using a deterministic mapping method, that is, the expert is directly assigned to the unique corresponding computing device to perform calculations.

[0050] For elasticity experts, since their corresponding set of mappable devices includes multiple candidate computing devices, the system does not need to immediately determine their execution device. Instead, it retains all mappable devices as a candidate device set. In subsequent scheduling, the system calculates a score for each candidate device based on its current load status and static bias, and selects the computing device with the highest score as the target device to achieve device-level load balancing scheduling.

[0051] After this step is completed, the system establishes two independent scheduling execution paths based on the expert type: a rigid expert deterministic scheduling path and a flexible expert dynamic equilibrium scheduling path. These two paths handle different types of experts respectively, and are ultimately merged during the result reorganization stage to form a complete expert routing result.

[0052] Step S300: Based on the unique deployment location of the rigid experts, assign the active rigid experts to the corresponding computing devices, and initialize the device load information based on the number of active rigid experts hosted by each computing device.

[0053] In one embodiment of the present invention, such as Figure 4 As shown, step S300 can be implemented in the following manner: Step S301: Calculate the number of deployment replicas of each expert based on the expert deployment matrix, and determine the active rigid experts in the current inference batch by combining the expert load vector.

[0054] The system reads the expert deployment matrix and expert load vector, where the expert deployment matrix represents the deployment relationship of each expert on each computing device, and can be represented as:

[0055] in, Indicates the number of computing devices. Indicates the number of experts. Experts Deployed on computing devices , Experts Not deployed on computing devices .

[0056] The expert load vector can be represented as:

[0057] in, This indicates the expert assigned to the current inference batch. of quantity.

[0058] The system sums the expert deployment matrix along the computing device dimension to obtain the number of deployment replicas for each expert:

[0059] in, Experts Number of deployed replicas across all computing devices.

[0060] When experts When the number of deployment replicas is equal to 1, it indicates that the expert is deployed on only one computing device, and can be identified as a rigid expert; when the expert When the number of deployed replicas is greater than 1, it indicates that the expert is deployed on multiple computing devices and can be identified as an elastic expert.

[0061] The system further determines the activity level of each expert in the current inference batch based on the expert load vector. The active identifier can be represented as:

[0062] in, Experts If a token to be processed is received in the current inference batch, the expert is considered an active expert. Experts If no token to be processed is received in the current inference batch, it will not participate in this routing process.

[0063] The system generates a rigid expert mask based on the number of deployed replicas and the activity status of the experts:

[0064] in, Indicates an indicator function. When experts When the number of deployed replicas is equal to 1 and the expert load value is greater than 0, Experts The active rigid expert in the current inference batch; otherwise, .

[0065] Step S302: Filter the expert deployment matrix according to the rigid expert mask to generate a rigid expert routing matrix.

[0066] The system extends the rigid expert mask to the dimension corresponding to the expert deployment matrix and multiplies it element-wise with the expert deployment matrix to obtain the rigid expert routing matrix:

[0067] in, Indicates active rigid experts in the current inference batch. With computing devices The routing relationship between them.

[0068] when At that time, it indicated that the experts For active rigid experts, and these experts are deployed on computing devices. ;when At that time, it indicated that the experts Not an active rigid expert, or an expert Not deployed on computing devices .

[0069] Since rigid experts have only one deployment copy, for any active rigid expert It has only one element with a value of 1 in the rigid expert routing matrix, that is:

[0070] Therefore, the rigid expert routing matrix can directly represent the unique computing device corresponding to each active rigid expert, without the need to perform additional target device search, device scoring, or candidate device comparison.

[0071] Step S303: Summate the rigid expert routing matrix along the expert dimension to obtain the number of active rigid experts supported by each computing device, and use this as the initial load information of the device.

[0072] For computing devices The corresponding initial device load can be expressed as:

[0073] Expanding the rigid expert routing matrix, it can be represented as:

[0074] in, Indicates computing device The number of active rigid experts already connected before elastic expert routing is implemented. The initial device load is an expert task counting load; for each active rigid expert connected, the corresponding device load increases by one unit.

[0075] For example, when computing devices The system has three rigid experts deployed, and if only two of these rigid experts have an expert load value greater than 0 in the current inference batch, the computing device... The initial device load is 2. Inactive rigid experts are not included in the initial device load.

[0076] The system saves the rigid expert routing matrix as the rigid expert routing portion of the complete expert routing matrix. Let the complete expert routing matrix be:

[0077] During the initialization of the complete expert routing matrix, the rigid expert routing matrix is ​​written into the complete expert routing matrix:

[0078] If an active elastic expert is subsequently found, the routing results of the elastic expert will be written into the complete expert routing matrix.

[0079] When no active elastic experts exist in the current inference batch, the rigid expert routing matrix constitutes the complete expert routing result for this inference. For the current computing device... The system reads the row corresponding to the current computing device from the complete expert routing matrix and multiplies it element-by-element with the expert load vector to obtain the actual number of expert tokens received by the current computing device.

[0080] in, Indicates computing device For experts The actual number of tokens that need to be processed.

[0081] The system further selects the number of tokens corresponding to the experts deployed on the current computing device from the number of tokens for each expert, based on the deployment relationship of the current computing device in the expert deployment matrix, to form the local expert load vector for the current computing device, which is used to perform subsequent expert inference calculations. It should be noted that the initial device load... With the number of expert tokens They have different meanings: the initial device load is used to indicate the number of active experts that the computing device has already handled, and serves as the load basis for subsequent elastic expert device selection; the number of expert tokens is used to indicate the actual amount of computing tasks that each computing device needs to process after routing is completed.

[0082] Step S400: For the elastic experts in the current inference batch, perform batch cascading routing based on the expert deployment matrix and the current device load information, determine the target computing device corresponding to each elastic expert, and update the device load information after each processing batch is completed.

[0083] For experts deployed on multiple computing devices, the system determines the range of mappable devices based on the expert deployment matrix, and selects the target computing device within the range of mappable devices based on the current device load status.

[0084] Let the expert deployment matrix be:

[0085] in, Indicates the number of computing devices. Indicates the number of experts; when At that time, it indicated that the experts Deployed on computing devices ;when At that time, it indicated that the experts Not deployed on computing devices .

[0086] Let the expert load vector corresponding to the current inference batch be:

[0087] in, This indicates the expert assigned to the current inference batch. The number of tokens.

[0088] expert The corresponding number of deployed replicas can be expressed as:

[0089] The system determines the active elastic experts in the current inference batch based on the number of deployed replicas and the expert load value. The elastic expert identifier can be represented as:

[0090] in, Indicates an indicator function. When experts... When deployed on two or more computing devices and their expert load value is greater than 0. Experts The active elasticity expert in the current inference batch.

[0091] The set of active elasticity experts can be represented as:

[0092] For any active elasticity expert The corresponding set of mappable devices is:

[0093] When the active elastic expert set is empty, the system skips the elastic expert device selection process and directly generates the expert token load vector corresponding to the current computing device based on the rigid expert routing results.

[0094] When the active elasticity expert set is not empty, such as Figure 5 As shown, step S400 can be achieved by following these steps.

[0095] Step S401: Extract the deployment relationships corresponding to active elastic experts based on the expert deployment matrix, and generate an elastic expert deployment sub-matrix.

[0096] The system extracts the corresponding columns from the expert deployment matrix according to the index of the active elastic experts in the expert set, and performs dimensional transformation to obtain the elastic expert deployment sub-matrix:

[0097] in, This indicates the number of active resilience experts. Indicates active elasticity expert With computing devices The deployment relationship between them.

[0098] when When, it indicates a computing device Deployed experts A copy can be used as a candidate target device for the expert; when When, it indicates a computing device No experts deployed Therefore, it cannot be used as the target equipment for this expert.

[0099] Step S402: Construct static reachability biases based on the elasticity expert deployment submatrix.

[0100] For each combination of active elastic experts and computing devices, the system generates a static reachability bias based on the corresponding deployment relationship, which is represented as follows:

[0101] in, This represents the preset path masking constant.

[0102] In one specific embodiment:

[0103] Therefore, the static reachability bias can be further expressed as:

[0104] For computing devices with deployed expert replicas, the static reachability bias is 0; for computing devices without deployed expert replicas, the static reachability bias is a negative masking value, placing the computing device at a very low priority in subsequent scoring processes.

[0105] Static reachability bias is used only to represent the reachability relationship between the expert and the computing device. In this embodiment, no random perturbation is added to the static reachability bias.

[0106] Step S403: Divide the active elastic experts into multiple processing batches arranged at intervals according to the preset number of processing steps.

[0107] Let the preset number of processing steps be The active elasticity experts are arranged in index order as follows:

[0108] For the One processing step, in which The current processing batch is formed by selecting elastic experts whose local indices satisfy the following formula:

[0109] Therefore, the processing intervals for different processing steps are as follows: The elasticity expert. Each processing batch sequentially performs scoring calculation, target device selection, and device load update.

[0110] Step S404: For each elastic expert in the current processing batch, calculate the device score based on the static reachability bias and the current device load information.

[0111] Before routing to elastic experts, the current device load state of each computing device is initialized by the routing results of active rigid experts.

[0112] For computing devices Its initial equipment load is:

[0113] in, Experts Is it an active rigid expert in the current inference batch?

[0114] The device load status indicates the number of active experts that the computing device has already connected with, rather than the number of tokens received by the computing device.

[0115] For the elastic experts in the current processing batch and computing devices The score is calculated as follows:

[0116] in, This indicates that the device load status value will be shifted left by 10 bits.

[0117] The above scoring formula is equivalent to:

[0118] in, Indicates static reachability bias. Indicates the computing device at the start of the current processing batch. The number of active experts already accepted.

[0119] For mappable computing devices Therefore, its score is mainly determined by the current device load status; the lower the device load, the higher the corresponding score. For non-mappable computing devices, Its score is negatively masked, preventing the device from being selected as the target device within the normal load range.

[0120] It should be noted that multiple elasticity experts in the same processing batch use the same initial device load status for parallel scoring. Once all target devices in the current batch are determined, the device load status is updated uniformly.

[0121] Step S405: Select the computing device with the highest score as the target computing device for the current elasticity expert.

[0122] For each elastic expert in the current processing batch The system compares the scores of all computing devices and selects the computing device with the highest score as the target computing device.

[0123] Because the computing device without an expert copy has a negative shielding bias, the selected target computing device satisfies:

[0124] Without setting random perturbations, when multiple mappable computing devices have the same maximum score value, the system determines the target computing device by the position of the first maximum value returned by the maximum value index operation.

[0125] The system records the selected target computing device number into the elastic expert target device index vector, which is used to generate the complete expert routing matrix later.

[0126] Step S406: Update the device load information of each computing device according to the elastic expert allocation results in the current processing batch.

[0127] After all elastic experts in the current processing batch have completed the target device selection, the system counts the number of elastic experts assigned to each computing device in the current processing batch and adds them to the current device load status of the corresponding computing device.

[0128] For computing device g, the device load update process can be represented as:

[0129] in, This represents the set of elastic experts in the current processing batch.

[0130] Each elasticity expert is assigned to a computing device The load status of the computing device increases by one unit. The update process does not use the number of tokens corresponding to the experts as the cumulative weight; that is, different active elastic experts contribute one load unit each during the device load update process.

[0131] After updating the device load for the current batch, the system continues processing the next batch. Subsequent batches calculate device scores based on the updated device load status, thus enabling cascading feedback of device load information between different batches.

[0132] The process continues until all processing batches are completed, at which point the target computing device indexes for all active elastic experts are obtained.

[0133] Step S500: Construct a complete expert routing matrix based on the rigid expert routing results and the elastic expert target device index.

[0134] After completing the rigid expert deterministic routing and the elastic expert batch cascaded routing, the system writes the device allocation results corresponding to the two types of experts into a unified expert routing matrix to represent the target computing device corresponding to each active expert in the current inference batch.

[0135] In one embodiment of the present invention, such as Figure 6 As shown, step S500 can be implemented in the following manner.

[0136] Step S501: Initialize the complete expert routing matrix.

[0137] The system first constructs a complete expert routing matrix:

[0138] in, Indicates the number of computing devices. The matrix represents the number of experts, with rows corresponding to computing devices and columns corresponding to experts.

[0139] All elements in the complete expert routing matrix are initialized to 0:

[0140] Matrix elements Used to indicate the experts in the current inference batch. Whether it is assigned to a computing device .when At that time, it indicated that the experts The current computing task is routed to the computing device. ;when At that time, it indicated that the experts Not routed to computing device .

[0141] Step S502: Write the routing results of the active rigid experts into the complete expert routing matrix.

[0142] The system writes the rigid expert routing matrix obtained in the preceding steps into the complete expert routing matrix.

[0143] Let the rigid expert routing matrix be:

[0144] in:

[0145] The system writes the rigid expert routing results according to the following formula:

[0146] Since the initial value of the complete expert routing matrix is ​​0, the above writing process is equivalent to:

[0147] For any active rigid expert, there is only one deployed copy, so the expert has only one element with a value of 1 in the corresponding column of the full expert routing matrix. For rigid experts that are not active in the current inference batch, their corresponding column remains 0.

[0148] Step S503: Write the target device index of the active elastic expert into the complete expert routing matrix.

[0149] For each active elastic expert, the system reads the target device index obtained from the aforementioned batch cascading routing process and writes a valid routing value at the corresponding position in the complete expert routing matrix.

[0150] Establish active elasticity experts The corresponding target computing device is The system then writes the elastic expert routing results according to the following formula:

[0151] For other computing device locations corresponding to the same active elastic expert, the matrix elements remain at 0.

[0152] Therefore, for any active elastic expert ,satisfy:

[0153] That is, each active elastic expert selects only one target computing device in the current inference batch to perform subsequent expert calculations.

[0154] After writing the routing results for both rigid and flexible experts, the complete expert routing matrix can be represented as:

[0155] The complete expert routing matrix uniformly represents the target device allocation relationship between active rigid experts and active elastic experts in the current inference batch. Inactive experts in the current inference batch do not generate routing tasks, and their corresponding columns remain 0.

[0156] It should be noted that this step is used to establish the routing relationship between experts and target computing devices, and does not recalculate or update device load in this step. Device load updates caused by elastic expert allocation have already been completed in the aforementioned batch cascading routing process.

[0157] Step S600: Generate the expert term load vector corresponding to the current computing device based on the complete expert routing matrix.

[0158] After obtaining the complete expert routing matrix, the system generates the number of expert tokens that the current computing device needs to process based on the routing results corresponding to the current computing device and the expert load vector, and further converts them into token load vectors arranged according to the local expert deployment order.

[0159] In one embodiment of the present invention, such as Figure 7 As shown, step S600 can be implemented in the following manner.

[0160] Step S601: Extract the expert routing mask corresponding to the current computing device.

[0161] Let the number of the computing device currently performing routing processing be . The system extracts computing devices from the complete expert routing matrix. Corresponding line:

[0162] Use the matrix rows as the expert routing mask corresponding to the current computing device:

[0163] Among them, when At that time, it indicated that the experts Routed to computing device in the current inference batch ;when At that time, it indicated that the experts Not routed to computing device .

[0164] When there are no active elastic experts in the current inference batch, the row corresponding to the current computing device can be directly extracted from the rigid expert routing matrix as the expert routing mask, which has the same meaning as the corresponding row of the complete expert routing matrix.

[0165] Step S602: Generate the current device term load vector arranged according to the global expert number. The system multiplies the expert routing mask corresponding to the current computing device element by element with the expert load vector to obtain the current device token load vector arranged according to the global expert number:

[0166] Therefore, we can conclude that:

[0167] in, This indicates element-wise multiplication.

[0168] When experts Routed to the current computing device hour, ,therefore:

[0169] When experts Not routed to the current computing device hour, ,therefore:

[0170] The length of the global expert index token payload vector is equal to the total number of experts. Each element corresponds to a global expert number, representing the number of tokens that the current computing device needs to process for each global expert.

[0171] Step S603: Generate local expert lexical load vectors based on the expert deployment relationships of the current computing devices. The system reads the current computing devices from the expert deployment matrix. Corresponding deployment relationships:

[0172] when At that time, it indicated that the experts Deployed on current computing devices ;when At that time, it indicated that the experts Not deployed on current computing devices .

[0173] Based on the positions of elements with a value of 1 in the expert deployment matrix, the system sequentially selects the corresponding elements from the global expert index token load vector to obtain the local expert token load vector of the current computing device:

[0174] Assuming deployment on current computing devices The local experts are arranged according to their global expert numbers as follows:

[0175] in, The number of local experts currently deployed on the computing device is represented by the local expert token load vector, which can then be expressed as:

[0176] It should be noted that for local experts deployed on the current computing device but not activated by routing in the current inference batch, their corresponding value in the local expert token payload vector is 0. Therefore, the local expert token payload vector maintains both the local expert arrangement order on the current computing device and reflects the actual number of tokens that each local expert needs to process in the current inference batch.

[0177] Step S604: Output the expert lexical load results corresponding to the current computing device.

[0178] The system outputs the following two results:

[0179] as well as:

[0180] in, The current device token payload vector, sorted by global expert ID; This is a local expert token load vector arranged according to the current local expert deployment order on the computing device.

[0181] The output results are used by the subsequent expert calculation module to organize the token inputs corresponding to each local expert on the current computing device. The subsequent expert calculation module can determine the number of tokens that each local expert needs to process based on the local expert token payload vector, and perform the corresponding expert forward calculation.

[0182] Compared with existing technical solutions, the multi-card expert balanced inference method proposed in this application, for hybrid expert models, achieves dynamic balanced allocation of expert computing resources during the inference stage by constructing a hierarchical scheduling mechanism that combines rigid and flexible experts, and combining static bias guidance and dynamic load-aware scheduling strategies. This method offers the following advantages: First, this application addresses the issue of uneven device load during the inference phase of hybrid expert models by proposing a scheduling method for device-level expert activation balancing. Existing MoE load balancing methods typically focus more on token-level distribution, achieving local balance by adjusting the routing relationship between tokens and experts. However, in low-concurrency, single-token, or sparse activation scenarios, it is easy for experts on some computing devices to be frequently accessed, while other computing devices remain idle, resulting in uneven distribution of computational pressure among devices.

[0183] This application analyzes the deployment relationships of experts on different computing devices and divides experts into rigid experts and elastic experts. Rigid experts are deterministically mapped based on unique deployment relationships, while elastic experts are dynamically selected based on a set of candidate devices. This improves the scheduling objective from traditional token-level balancing to device-level balancing of expert activation times, thereby reducing the computing congestion caused by concentrated access by hot experts on a single device and improving the overall resource utilization of multi-GPU systems.

[0184] Secondly, this application addresses the high complexity of combinatorial optimization in multi-GPU MoE expert scheduling by proposing an approximate optimization scheduling method based on a load-aware scoring function. Since the mapping relationship between experts and computing devices is influenced by multiple factors such as expert deployment constraints, device resource status, and expert replica distribution, it is essentially a constrained combinatorial optimization problem. Solving this problem using integer programming (ILP) or global search methods requires traversing a large number of possible expert-device combinations, which introduces high scheduling overhead in AI accelerator inference scenarios, making it difficult to meet real-time inference requirements.

[0185] This application utilizes static bias terms to describe the structural relationship between experts and devices, and dynamic load terms to reflect the current resource occupancy of devices. Expert allocation is completed by progressively selecting the candidate device with the highest score, transforming the complex combinatorial optimization problem into a low-complexity local extremum selection process. While ensuring scheduling effectiveness, it reduces computational complexity, making it more suitable for online scheduling needs during large-scale MoE model inference.

[0186] Furthermore, this application achieves hierarchical processing of the scheduling process by combining deterministic scheduling of rigid experts with dynamic load balancing scheduling of elastic experts. For rigid experts with only a single deployed device, since their mapping relationship is uniquely determined by the expert deployment matrix, no additional search process is required, and the deterministic mapping from expert to computing device is directly completed. For elastic experts with multiple replicas, dynamic selection is performed based on the candidate device set, static bias, and real-time load status. By decoupling the deterministic mapping path from the optimized scheduling path, the additional computational overhead caused by all experts uniformly entering a complex optimization process is avoided, while improving the overall scheduling efficiency.

[0187] Meanwhile, this application employs a dynamic feedback mechanism based on load-based recursive updates during the elastic expert scheduling process. After selecting a device for an expert, the load status of the corresponding device is immediately updated, enabling subsequent expert scheduling processes to be aware of the latest device load distribution and dynamically adjust subsequent selection results. Compared to a fixed expert mapping method, this application can automatically correct load deviations between devices as the inference process changes, gradually balancing the number of active experts across multiple computing devices. This reduces waiting time caused by overloaded hotspot devices and improves inference throughput.

[0188] Furthermore, this application combines batch-level parallel scheduling to transform the traditional expert allocation process, which requires global solution, into a lightweight computational process that can be executed in parallel. Rigid experts can be directly mapped based on the deployment matrix, while elastic experts can calculate candidate device scores in parallel based on pre-calculated static biases and current load states, thereby reducing the synchronization waiting overhead during scheduling. Simultaneously, this method relies only on structured information such as the expert deployment matrix, expert load vectors, and device load states, without introducing additional complex control modules. Therefore, it has good hardware adaptability and can be deployed on multi-GPU AI acceleration platforms such as NPUs and GPUs.

[0189] To verify the effectiveness of the technical solution proposed in this application, inference latency tests were conducted on the proposed method, the Baseline method, and the Deepseek-AI solution during the inference stage of the Qwen3-Next model. The test results are shown in Table 1 below:

[0190] Further analysis of the experimental results reveals that as the batch size increases, the Baseline method, lacking an expert scheduling mechanism for device-level load balancing, easily leads to computational hotspots on some devices, resulting in a more significant increase in inference latency. While the Deepseek-AI scheme employs some expert scheduling optimization, it primarily relies on static or weakly dynamic strategies, making it difficult to continuously adjust the expert activation distribution among devices under high load conditions. The method in this application, through a dynamic scheduling mechanism combining rigid and flexible experts, enables device load to adjust in real time during the expert allocation process, thus maintaining a low inference latency growth trend across different batch sizes.

[0191] In summary, the multi-GPU expert balanced inference method for hybrid expert models proposed in this application achieves efficient and balanced allocation of expert computing resources during the MoE model inference stage through techniques such as expert classification scheduling, static bias guidance, dynamic load awareness, and result fusion output. Compared with existing solutions, this application can effectively reduce computational waiting caused by uneven load between devices, improve the resource utilization of multi-GPU AI acceleration platforms, and reduce computational complexity and overhead in the expert scheduling process. It is suitable for the efficient deployment of large-scale hybrid expert models in multi-GPU inference scenarios.

[0192] Figure 8 This is a schematic diagram of the structure of a multi-card expert equilibrium inference system for hybrid expert models provided in an embodiment of the present invention. This system is applied in the inference phase, such as... Figure 8 As shown, it includes the following modules: Input processing module 1 is used to obtain the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert.

[0193] Expert classification module 2 is used to identify experts whose set of mappable devices is 1 as rigid experts and experts whose set of mappable devices is greater than 1 as flexible experts.

[0194] The rigid expert allocation module 3 is used to allocate active rigid experts to corresponding computing devices based on the unique deployment location of the rigid experts, and initialize device load information based on the number of active rigid experts hosted by each computing device.

[0195] Elastic expert allocation module 4 is used to perform batch cascading routing for elastic experts in the current inference batch based on the expert deployment matrix and the current device load information, determine the target computing device corresponding to each elastic expert, and update the device load information after each processing batch is completed.

[0196] The expert routing matrix construction module 5 is used to construct a complete expert routing matrix based on the rigid expert routing results and the elastic expert target calculation device index.

[0197] The expert terminator load vector generation module 6 is used to generate the expert terminator load vector corresponding to the current computing device based on the complete expert routing matrix.

[0198] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-card expert equilibrium inference method for hybrid expert models, characterized in that, Applied to the reasoning stage, including: Obtain the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert; Experts with a set of mappable devices of size 1 are defined as rigid experts, and experts with a set of mappable devices of size greater than 1 are defined as elastic experts. Based on the unique deployment location of the rigid experts, the active rigid experts are assigned to the corresponding computing devices, and the device load information is initialized based on the number of active rigid experts handled by each computing device. For the elastic experts in the current inference batch, batch cascading routing is performed based on the expert deployment matrix and the current device load information to determine the target computing device corresponding to each elastic expert, and the device load information is updated after each processing batch is completed. Construct a complete expert routing matrix based on the rigid expert routing results and the elastic expert target calculation device index; Generate the expert term load vector corresponding to the current computing device based on the complete expert routing matrix.

2. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The process of obtaining the expert deployment matrix and expert load vector, and determining the set of mappable devices for each expert, includes: Obtain an expert deployment matrix to characterize the deployment relationships of experts on different computing devices; Obtain the expert load vector, which represents the load distribution information of each expert during the reasoning process; The set of mappable devices corresponding to each expert is determined based on the expert deployment matrix.

3. The multi-card expert equilibrium inference method for hybrid expert models according to claim 2, characterized in that, The step of obtaining the expert load vector, which characterizes the load distribution information of each expert during the inference process, includes: Obtain the Top-k expert gating results output by the hybrid expert model; The number of tokens assigned to each expert is calculated based on the Top-k expert gating results. The number of terms corresponding to each expert is used to construct the expert load vector.

4. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The method further includes: Convert the data format of the expert deployment matrix and expert load vector to INT32 type.

5. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The process of allocating active rigid experts to corresponding computing devices based on their unique deployment locations, and initializing device load information based on the number of active rigid experts handled by each computing device, includes: The number of deployment replicas for each expert is calculated based on the expert deployment matrix, and the active rigid experts in the current inference batch are determined by combining the expert load vector. The expert deployment matrix is ​​filtered based on the rigid expert mask to generate a rigid expert routing matrix; The rigid expert routing matrix is ​​summed along the expert dimension to obtain the number of active rigid experts supported by each computing device, which is then used as the initial load information for the device.

6. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The process involves performing batch cascading routing for elastic experts in the current inference batch based on the expert deployment matrix and current device load information, determining the target computing device corresponding to each elastic expert, and updating the device load information after each processing batch is completed. This includes: Based on the expert deployment matrix, extract the deployment relationships corresponding to active elastic experts and generate an elastic expert deployment submatrix; Static reachability biases are constructed based on the elasticity expert deployment submatrix. Active elastic experts are divided into multiple processing batches arranged at intervals according to the preset number of processing steps; For each elasticity expert in the current processing batch, calculate the device score based on the static reachability bias and the current device load information; Select the computing device with the highest score as the target computing device for the current elasticity expert; Update the device load information of each computing device based on the elastic expert allocation results in the current processing batch.

7. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The construction of the complete expert routing matrix based on the rigid expert routing results and the elastic expert target computing device index includes: Initialize the complete expert routing matrix; Write the routing results of the active rigid experts into the complete expert routing matrix; Write the target device index of the active elastic expert into the complete expert routing matrix.

8. The multi-card expert equilibrium inference method for hybrid expert models according to claim 1, characterized in that, The step of generating the expert lexical load vector corresponding to the current computing device based on the complete expert routing matrix includes: Extract the expert routing mask corresponding to the current computing device; Generate the current device term load vector arranged according to the global expert ID; Generate local expert terminology payload vectors based on the expert deployment relationships of the current computing devices; Output the expert lexical load results corresponding to the current computing device.

9. A multi-card expert equilibrium inference system for hybrid expert models, characterized in that, Applied to the reasoning stage, including: The input processing module is configured to acquire the expert deployment matrix and expert load vector, and determine the set of mappable devices for each expert; The expert classification module is configured to identify experts whose set of mappable devices is 1 as rigid experts and experts whose set of mappable devices is greater than 1 as flexible experts. The rigid expert allocation module is configured to allocate active rigid experts to corresponding computing devices based on the unique deployment location of the rigid experts, and initialize device load information based on the number of active rigid experts hosted by each computing device. The Elastic Expert Allocation Module is configured to perform batch cascading routing for the elastic experts in the current inference batch based on the expert deployment matrix and the current device load information, determine the target computing device corresponding to each elastic expert, and update the device load information after each processing batch is completed. The expert routing matrix construction module is configured to construct a complete expert routing matrix based on rigid expert routing results and elastic expert target calculation device indexes. The expert terminology load vector generation module is configured to generate the expert terminology load vector corresponding to the current computing device based on the complete expert routing matrix.

Images