Method and system for multi-device load balancing of mixed expert large language model inference
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SEMICON TECH INNOVATION CENT(BEIJING) CORP
- Filing Date
- 2025-12-09
- Publication Date
- 2026-06-05
AI Technical Summary
In existing parallel inference schemes for hybrid expert models (MoE), device load imbalance leads to idle and overloaded computing resources, and the amount of cross-device data transmission increases. This cannot meet the latency stability requirements of industrial scenarios and is difficult to adapt to the dynamic sparsity characteristics of the MoE model, thus becoming an obstacle restricting its large-scale application.
A reference dataset of expert activation features is constructed. Through initial grouping and iterative optimization strategies, combined with the number and parallelism of the device cluster, the expert storage allocation is optimized to achieve balanced device load.
It significantly improves the utilization of computing resources, reduces the risk of equipment failure, and enhances system stability and deployment adaptability. It is suitable for homogeneous, heterogeneous, and three-dimensional stacked devices and supports efficient load balancing for small-to-medium-sized to ultra-large-scale clusters.
Smart Images
Figure CN122152494A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-device load balancing method and system for hybrid expert large language model reasoning. Background Technology
[0002] Mixture of Experts (MoE) models, as an innovative architecture that balances model size and computational efficiency, are based on the core design concept of "sparse activation." Because sparse activation requires invoking only a subset of experts for each inference iteration, computational cost becomes sublinearly related to the number of parameters, significantly improving efficiency. This has led to the widespread application of MoE in ultra-large-scale language models and multimodal models.
[0003] In existing parallel inference schemes for Hybrid Expert Models (MoE), expert parallelism is the mainstream deployment paradigm. Its core is to statically split a large number of expert networks across different computing devices (such as GPUs) or computing cores according to a fixed strategy. Each device is responsible for storing and running several expert networks assigned to it. The initial intention of this approach was to match the computational demands brought about by the surge in the number of experts in the MoE model through the parallelization of hardware resources, theoretically allowing for linear expansion of model capacity with the increase in the number of devices. However, this static splitting strategy fundamentally contradicts the dynamic activation characteristics of MoE: the gating network's selection of experts has a strong input dependency and real-time dynamism. Differences in the features of different input samples can lead to drastic changes in the gating weight distribution, thereby activating completely different expert combinations. This results in unpredictable dynamic fluctuations in device load. This dynamic load imbalance triggers a series of chain reactions: the computing power and memory resources of idle devices remain idle for extended periods, contradicting MoE's design intent of "improving efficiency through sparse activation"; when activated experts are distributed across multiple devices, input data needs to be frequently migrated between devices (e.g., distributing the same input sample to multiple expert devices), and the amount of cross-device data transfer increases exponentially with the degree of load dispersion, becoming a new performance bottleneck; moreover, load fluctuations cause drastic changes in device computational pressure, with inference latency jitter reaching several times or even tens of times, failing to meet the latency stability requirements of industrial-grade scenarios. Therefore, existing expert parallel solutions are difficult to adapt to the dynamic sparsity characteristics of MoE, resulting in the actual efficiency of its distributed deployment being far lower than theoretical expectations, becoming a core obstacle restricting the large-scale industrial application of MoE models. Summary of the Invention
[0004] This invention provides a high-efficiency and stable multi-device load balancing method and system for hybrid expert large language model inference, which can significantly improve the utilization of computing resources and facilitate the industrial application of distributed inference of hybrid expert large language models.
[0005] This invention provides a multi-device load balancing method for hybrid expert large language model inference, comprising:
[0006] Construct a reference dataset of expert activation features to form expert activation frequency distribution data covering multiple preset task scenarios;
[0007] Based on the number of target device clusters and the preset parallelism, all experts are initially grouped to form an initial storage allocation scheme.
[0008] For the initial grouping and reference dataset, the initial latency of the target device cluster is calculated using a quantitative evaluation mechanism, which considers factors including the computational latency caused by the computation time of activating experts.
[0009] An iterative optimization strategy is adopted by exchanging experts in pairs between groups to ensure that the final latency reaches the preset range and obtain the final expert storage allocation scheme.
[0010] This invention also provides a multi-device load balancing system for hybrid expert large language model inference, comprising:
[0011] The dataset construction module is used to build a reference dataset of expert activation features, forming expert activation frequency distribution data covering multiple preset task scenarios;
[0012] The initial grouping module is used to initially group all experts based on the number of target device clusters and the preset parallelism, thereby forming an initial storage allocation scheme.
[0013] The initial latency calculation module is used to calculate the initial latency of the target device cluster using a quantitative evaluation mechanism for the initial grouping and reference dataset. The quantitative evaluation mechanism takes into account factors including the computational latency caused by the computation time of activating experts.
[0014] The iterative optimization module is used to perform an iterative optimization strategy by exchanging experts in pairs between groups, so that the final delay reaches the preset range and the final expert storage allocation scheme is obtained.
[0015] This invention constructs an expert activation feature reference dataset and combines iterative optimization strategies to achieve precise matching between expert storage and actual activation frequency, fundamentally avoiding device idleness caused by concentrated expert activation. Furthermore, through "device-expert" performance mapping and comprehensive load balancing evaluation, a dynamic balance is achieved where "high-activation-frequency experts are matched with high-performance devices, and low-activation-frequency experts are adapted to low-performance devices," preventing low-performance devices from becoming global bottlenecks due to high loads. By using offline activation statistics and iterative optimization, most optimization logic is moved to the "storage allocation stage," eliminating the need for additional dynamic scheduling during inference; calculations are only performed according to the pre-optimized storage scheme, reducing deployment complexity by over 60%. Simultaneously, improved load balancing reduces the risk of hardware failures due to long-term overload (such as GPU memory overflow and CPU overheating), improving system continuous operation stability by 50% and reducing maintenance costs. Moreover, the optimization strategy of this invention can seamlessly adapt to various hardware architectures, including homogeneous devices, heterogeneous devices, and 3D stacked devices, without requiring modification to the model's network structure or inference logic. Whether it's a small-to-medium-sized cluster (e.g., 8-16 devices) or a massively distributed system (e.g., 100+ devices), efficient optimization can be achieved by adjusting the reference dataset and iteration rounds, providing flexible technical support for the deployment of MoE-LLM in various scenarios such as cloud and edge computing. Therefore, this invention solves the core load balancing problem of parallel inference in MoE-LLM from the storage allocation level by deeply coupling expert activation features with device characteristics (homogeneous / heterogeneous). It achieves breakthroughs in resource utilization, inference efficiency, system stability, and deployment adaptability, laying a key technical foundation for the large-scale industrial application of hybrid expert models. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 A flowchart of a multi-device load balancing method for hybrid expert large language model inference provided in an embodiment of the present invention;
[0018] Figure 2 for Figure 1 The flowchart of load balancing optimization for homogeneous devices in the illustrated embodiment is shown.
[0019] Figure 3 for Figure 1 The flowchart of load balancing optimization for heterogeneous devices in the embodiment shown is as follows;
[0020] Figure 4 This invention provides a schematic diagram of the structure of a multi-device load balancing system with hybrid expert large language model inference.
[0021] Figure 5 This is a schematic diagram illustrating an application example of the present invention in a homogeneous multi-device architecture. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0023] Mixture of expert models The "sparse activation" mechanism in Experts (MoE) replaces the feedforward neural network (FFN) in a Large Language Model (LLM) based on the Transformer architecture with a hybrid layer containing multiple parallel "expert networks" and a "gated network." Each expert network is an independent sub-network of the FFN, focusing on processing a specific type of input pattern; the gated network dynamically calculates the weight distribution based on the features of the input samples, selecting only a few of the most relevant expert networks to participate in the computation, while the rest remain "dormant." The core advantage of this design is that by increasing the number of experts, the number of parameters can be significantly expanded without significantly increasing the computational cost per inference; and the sparse activation mechanism ensures that only a portion of the experts are invoked in each inference, making the computational cost sublinear with the number of parameters, significantly improving efficiency. However, the sparsity of MoE also brings new challenges to parallel computing, the most prominent being the device load balancing problem. To this end, the multi-device load balancing method and system for hybrid expert large language model inference provided in this invention uses an iterative optimization strategy to balance the average expert load of different devices, so that the expert activation frequency is deeply coupled with the device load of storage allocation, avoiding the problem of device idleness or overload caused by concentrated expert activation, which is beneficial to the resource utilization and stability of MoE-LLM parallel inference.
[0024] To make the technical solution of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0025] Figure 1 This is a flowchart of a multi-device load balancing method for hybrid expert large language model inference provided in an embodiment of the present invention. Figure 2 for Figure 1 The illustrated embodiment shows a flowchart of load balancing optimization for homogeneous devices. Figure 3 for Figure 1 The flowchart of load balancing optimization for heterogeneous devices in the illustrated embodiment is as follows: Figure 1-3 As shown, the method in this embodiment includes:
[0026] Step 101: Construct a reference dataset of expert activation features to form expert activation frequency distribution data covering multiple preset task scenarios.
[0027] In this step, the activation frequency distribution data covering multiple scenarios is generated by statistically analyzing the expert activation records of the neural network layers of the hybrid expert model in the training set, test set, and various preset specific task scenarios. This data serves as the basis for subsequent optimization. Since the frequency of expert activation varies greatly across different task scenarios, while the frequency is similar within the same task scenario, the expert activation frequency in specific task scenarios is statistically analyzed beforehand to serve as a reference for subsequent expert activation frequencies in the same task scenarios.
[0028] In addition to the reference dataset built based on expert activation frequency, the heterogeneity of the devices also needs to consider the computation time of each device in processing each expert. Thus, the reference dataset also includes the maximum computation latency of a single device in the cluster for reference in subsequent quantitative evaluation mechanisms. That is, to build a "device-expert" performance mapping profile - quantify the computation time of each device in processing each expert through benchmark testing (such as pre-calculating the single inference time t_ij of device i running expert j), and combine it with the activation records of the MoE layer in the training set, test set and typical tasks to form a two-dimensional reference dataset containing "expert activation frequency-device processing time", which provides an accurate basis for load allocation.
[0029] Step 102: Based on the number of target device clusters and the preset parallelism, all experts are initially grouped to form an initial storage allocation scheme.
[0030] In this step, the initial grouping differs depending on whether it's based on homogeneous or heterogeneous devices. When the target device cluster consists of homogeneous multiple devices or homogeneous 3D stacked devices, the computation time for a single expert is the same on any device due to the homogeneity. Therefore, during initial grouping, each group of experts can be assigned an equal number of devices, or a single device can be responsible for expert computation within a specific group, forming the initial storage allocation scheme. When the initial grouping is based on heterogeneous multiple devices or heterogeneous 3D stacked devices, the energy efficiency of different devices varies due to their heterogeneity. In a heterogeneous environment, the load imbalance caused by the performance differentiation of devices and the dynamic activation of experts is more severe. Therefore, the energy efficiency differences of devices must also be considered during initial grouping. That is, during initial grouping, a load threshold is allocated to each device based on the energy efficiency differences (e.g., high-performance devices can handle more experts, while low-efficiency devices are assigned fewer experts) to ensure that the number of experts handled by high-performance devices is greater than the number handled by low-efficiency devices, ensuring that the initial grouping theoretically matches the processing capabilities of each device. Heterogeneous multiple devices can include GPUs, CPUs, or dedicated acceleration chips with different architectures.
[0031] Step 103: For the initial group and reference dataset, calculate the initial latency of the target device cluster using a quantitative evaluation mechanism. The quantitative evaluation mechanism considers factors including the computational latency caused by the computation time of activating experts.
[0032] Building upon the steps outlined above, a quantitative evaluation mechanism based on computational latency is introduced: For the initial grouping, the average computational latency of the device cluster is calculated (referred to as the baseline score) by combining the expert activation frequency in the reference dataset. This score comprehensively reflects the device load distribution of activation requests within the current group. To further optimize balance, an iterative optimization strategy is employed: The pairwise exchange operation of experts between groups is performed repeatedly, and the average latency (referred to as the optimization score) is recalculated for each exchange-based grouping scheme. If the optimization score is lower than the baseline score, it indicates that the device load is more balanced after the exchange. In this case, the grouping scheme is updated, and the optimization score is set as the new baseline. Through multiple iterations until the scores converge, the optimal expert storage allocation scheme is finally obtained, which can balance the average expert computational load of different devices.
[0033] For heterogeneous devices, the evaluation mechanism uses a "comprehensive load balancing score" instead of a single average latency metric. This score includes two core metrics: "cluster average computing latency" and "maximum computing latency of a single device," with the maximum latency having a higher weight (e.g., 60%) to mitigate the risk of low-performance devices bearing high loads. For example, if a solution has a low average latency but the latency of a single low-performance device is five times that of the others, its comprehensive score will be significantly lower. Therefore, in the quantitative evaluation mechanism, the maximum computing latency of a single device can be given a greater weight relative to the cluster's average latency.
[0034] Step 104: Adopt an iterative optimization strategy of pairwise expert exchanges between groups to ensure that the final latency reaches the preset range and obtain the final expert storage allocation scheme.
[0035] The initial latency, final latency, and latency calculated in each round in this step correspond to the initial average latency, final average latency, and average latency of the target device cluster in that round.
[0036] After completing the above steps to build the reference dataset and initial grouping, and selecting the quantitative evaluation mechanism, iterative optimization operations can be performed to find the best expert storage allocation scheme.
[0037] When the target device cluster consists of homogeneous multiple devices or homogeneous 3D stacked devices, the process iterates through the pairwise exchange operations of experts between groups. A quantitative evaluation mechanism recalculates the latency of the grouping scheme after each exchange. If the recalculated latency is less than the latency before the current exchange, the latency record is updated, and the grouping scheme after that exchange is used as the new expert storage allocation scheme. This process continues until the calculated latency reaches a preset range, at which point the iterative exchange operation ends, and the expert storage allocation scheme after the last round of exchanges is taken as the final expert storage allocation scheme. To ensure rapid convergence of the iteration, the preset range for the final latency can be set to 10% of the theoretical value.
[0038] When the target device cluster consists of heterogeneous multi-device or heterogeneous 3D stacked devices, the process iterates through the pairwise expert swap operations between groups. The latency is recalculated for each swapped grouping scheme. If, after the swap, a high-activity-frequency expert migrates from a low-performance device to a high-performance device, causing a decrease in maximum latency and the average latency not to reach a set increase threshold, then the grouping scheme after the swap is used as the new expert storage allocation scheme. This process continues until the cluster's average latency is within a preset range, and the maximum computational latency of a single device is also within a preset range. The iterative swapping operation ends, and the expert storage allocation scheme after the last round of swapping is used as the final expert storage allocation scheme. In other words, during the iterative optimization phase, when performing pairwise expert swaps between groups, not only is the change in average latency after the swap calculated, but the focus is also on evaluating whether the cluster's maximum latency has been reduced. If, after the swap, a high-activity-frequency expert migrates from a low-performance device to a high-performance device, causing a decrease in maximum latency and no significant increase in average latency, then it is considered an effective optimization, the grouping scheme is updated, and the overall score is set as the new benchmark. Through multiple rounds of iteration until the comprehensive score converges, the final storage allocation scheme can achieve a dynamic balance of "high activation frequency experts matching high-performance devices and low activation frequency experts adapting to low-performance devices", which reduces the overall latency of the cluster and avoids a single device becoming a performance bottleneck.
[0039] This invention constructs an expert activation feature reference dataset and combines iterative optimization strategies to achieve precise matching between expert storage and actual activation frequency, fundamentally avoiding the problem of device idleness caused by concentrated expert activation. For example, in a scenario with 64 experts and 8 devices deployed, traditional static allocation may result in a single device having a 100% load while the remaining devices are idle, with a resource utilization rate of only 12.5%. However, this invention, through activation statistics and group optimization, can control device load fluctuations within ±5%, increasing resource utilization to over 90%, and reducing hardware investment by 20%-30% under the same inference requirements. Moreover, through "device-expert" performance mapping and comprehensive load balancing evaluation, a dynamic balance is achieved where "high-activation-frequency experts are matched with high-performance devices, and low-activation-frequency experts are adapted to low-performance devices," preventing low-performance devices from becoming global bottlenecks due to high loads. By using offline activation statistics and iterative optimization, most optimization logic is moved forward to the "storage allocation stage," eliminating the need for additional dynamic scheduling during inference; calculations are only performed according to the pre-optimized storage scheme, reducing deployment complexity by over 60%. Meanwhile, improved load balancing reduces the risk of hardware failures caused by long-term overload (such as GPU memory overflow and CPU overheating), improves system stability by 50%, and reduces maintenance costs. Furthermore, the optimization strategy of this invention can seamlessly adapt to various hardware architectures, including homogeneous devices, heterogeneous devices, and 3D stacked devices, without requiring modification to the model's network structure or inference logic. Whether it's a small-to-medium-sized cluster (e.g., 8-16 devices) or a large-scale distributed system (e.g., 100+ devices), efficient optimization can be achieved by adjusting the reference dataset and iteration rounds, providing flexible technical support for the deployment of MoE-LLM in multiple scenarios such as cloud and edge computing. Therefore, this invention solves the core load balancing problem of MoE-LLM parallel inference from the storage allocation level by deeply coupling expert activation features with device characteristics (homogeneous / heterogeneous), achieving breakthroughs in resource utilization, inference efficiency, system stability, and deployment adaptability, laying a key technical foundation for the large-scale industrial application of hybrid expert models.
[0040] Figure 4 This invention provides a schematic diagram of the structure of a multi-device load balancing system with hybrid expert large language model inference, as shown in the embodiment of the invention. Figure 4As shown, the system in this embodiment includes: a dataset construction module 100, an initial grouping module 200, an initial latency calculation module 300, and an iterative optimization module 400. The dataset construction module 100 is used to construct a reference dataset of expert activation features, forming expert activation frequency distribution data covering multiple preset task scenarios. The initial grouping module 200 is used to initially group all experts based on the number of target device clusters and a preset parallelism, forming an initial storage allocation scheme. The initial latency calculation module 300 is used to calculate the initial latency of the target device cluster using a quantitative evaluation mechanism based on the initial grouping and the reference dataset. The quantitative evaluation mechanism considers factors including the computational latency caused by the computation time of activated experts. The iterative optimization module 400 uses a pairwise exchange strategy between experts in each group to ensure the final latency reaches a preset range, thereby obtaining the final expert storage allocation scheme.
[0041] The multi-device load balancing system for hybrid expert large language model inference in this embodiment can perform the above-mentioned... Figures 1-3 The solutions in any of the embodiments shown achieve similar technical effects, and will not be described in detail here.
[0042] The technical solution of the present invention and its achieved technical effects will be described below using actual application scenarios.
[0043] Figure 5 This is a schematic diagram illustrating an application example of the present invention in a homogeneous multi-device architecture, as shown below. Figure 5 As shown, this example addresses a homogeneous multi-device scenario, employing a computing cluster composed of four A6000 graphics cards to deploy the Deepseek-MoE model for parallel inference. The Deepseek-MoE model comprises multiple MoE layers, each with 64 expert networks. The gated network activates 8 experts per inference iteration. Since the A6000 graphics cards have consistent performance (homogeneous devices), the expert storage optimization strategy based on activation features, as described in this invention, is needed to address the load concentration problem caused by static partitioning, thereby improving device utilization and inference stability.
[0044] The specific implementation steps are as follows:
[0045] 1. Construct an expert activation feature reference dataset
[0046] We selected 100,000 data points from the training set, test set, and input samples from three typical task scenarios (text generation, question answering, and summarization) of the Deepseek-MoE model.
[0047] The model is run for inference, and the activation frequency of each of the 64 experts in each layer is recorded. The activation frequency distribution of each expert is then statistically analyzed. For example, experts E1-E10 have an activation frequency of over 60%, classifying them as high-frequency activation experts; while experts E51-E64 have an activation frequency of less than 5%, classifying them as low-frequency activation experts.
[0048] The activation frequency data was organized into a structured table, which served as the basis for subsequent grouping optimization.
[0049] 2. Initial Simulation Grouping
[0050] The target device consists of four A6000 graphics cards (denoted as GPU0-GPU3), with the preset parallelism matching the number of devices (i.e., each graphics card is responsible for a group of experts).
[0051] Based on the principle of evenly distributing the total number of experts, the 64 experts were initially divided into 4 groups of 16 experts each, corresponding to GPU0-GPU3 respectively. For example, GPU0 was assigned E1-E16, GPU1 was assigned E17-E32, and so on.
[0052] 3. Quantitatively assess the computational delay of the initial grouping.
[0053] Based on the activation frequency in the reference dataset, the average computational latency of the devices in the initial group is calculated: Assuming the time taken for a single expert computation is t (since the A6000 is isomorphic, the time taken for a single expert is consistent), the average number of activated experts for each graphics card is counted (e.g., GPU0 contains multiple high-frequency experts, so the average number of activated experts is 6; the average number of activated experts for GPU1 and GPU2 is 5 and 3 respectively, and GPU3 contains low-frequency experts, so the average number of activated experts is 2), and the baseline score is obtained (average latency = (6t + 5t + 3t + 2t) / 4 = 4t).
[0054] 4. Iterative optimization of grouping
[0055] Perform a pairwise expert swap between groups: for example, swap the high-frequency expert E5 in GPU0 with the low-frequency expert E58 in GPU3, and recalculate the average latency. If the average latency drops to 3.8t after the swap (the optimization score is lower than the baseline score), update the grouping scheme and set 3.8t as the new baseline.
[0056] Repeat the above swapping operation, traversing all possible inter-group expert combinations. After 20 rounds of iteration, the score converges to 3.2t. At this point, the grouping scheme is: the average number of activated experts per graphics card is stable at 4±0.2, and the load fluctuation is controlled within 5%.
[0057] 5. Deployment and Verification
[0058] According to the optimized grouping scheme, the expert parameters are deployed to 4 A6000 graphics cards: high-frequency experts are evenly distributed across the graphics cards to avoid concentrated load.
[0059] 10,000 test data points were selected for inference verification: the results showed that the average utilization of each graphics card increased from 30%-40% in traditional static allocation to 92%, the inference latency jitter decreased from ±50% to ±8%, and there was no single device overload phenomenon.
[0060] For the above application scenarios, the Deepseek-MoE model deployed on four A6000 graphics cards using the load balancing optimization strategy of this invention solves the load imbalance problem of traditional expert parallel solutions, realizes efficient parallel inference in homogeneous device scenarios, and achieves industrial-grade deployment requirements in terms of resource utilization and stability.
[0061] When the equalization optimization method of this invention is applied to a three-dimensional stacked device, such as deploying the OLMoE model on a three-dimensional stacked architecture system with 64 NPU cores, the 64-core NPU (Neural-network Processing Unit) in the three-dimensional stacked architecture system directly accesses the DRAM bank corresponding to its core through vertical hybrid bonding technology and cannot access other DRAM banks. Therefore, each NPU core can be regarded as a single device. The OLMoE model has 64 experts, with 8 activated at a time. If one expert corresponds to one NPU core, only 8 NPU cores will work at a time, leaving the remaining 56 NPU cores idle, resulting in a severely uneven NPU load. Therefore, in this architecture, this invention treats every 8 NPU cores as a processing tile group and processes one expert parameter using tensor parallelism. Thus, the 64-core NPU can be divided into 8 device groups. The 64 experts of the OLMoE model need to be divided into 8 groups of 8, with each group having 8 experts, and deployed for computation on one device group. Given the expert dynamic activation characteristics of the MoE model, in order to avoid load imbalance among the eight device groups, after the device groups are grouped, the optimization strategy of this invention can be used to achieve multi-core load balancing optimization under the three-dimensional stacked architecture.
[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-device load balancing method using hybrid expert large language model reasoning, characterized in that, include: Construct a reference dataset of expert activation features to form expert activation frequency distribution data covering multiple preset task scenarios; Based on the number of target device clusters and the preset parallelism, all experts are initially grouped to form an initial storage allocation scheme. For the initial grouping and reference dataset, the initial latency of the target device cluster is calculated using a quantitative evaluation mechanism, which considers factors including the computational latency caused by the computation time of activating experts. An iterative optimization strategy is adopted by exchanging experts in pairs between groups to ensure that the final latency reaches the preset range and obtain the final expert storage allocation scheme.
2. The method according to claim 1, characterized in that, If the target device cluster consists of homogeneous multiple devices or homogeneous three-dimensional stacked devices, an iterative optimization strategy of pairwise expert exchange between groups is adopted to achieve the preset convergence of latency and obtain the final expert storage allocation scheme, specifically: The process iterates through the pairwise exchange operations between experts in each group, using a quantitative evaluation mechanism to recalculate the latency of the grouping scheme after each exchange. If the recalculated latency is less than the latency before the current exchange, the latency record is updated, and the grouping scheme after this exchange operation is used as the new expert storage allocation scheme. This process continues until the calculated latency reaches a preset range, at which point the iterative exchange operation ends, and the expert storage allocation scheme after the last round of exchange operations is used as the final expert storage allocation scheme.
3. The method according to claim 2, characterized in that, The initial grouping is as follows: each group of experts corresponds to an equal number of devices or a single device is responsible for expert calculations within a specific group.
4. The method according to claim 2, characterized in that, The initial latency, final latency, and latency calculated in each round correspond to the initial average latency, final average latency, and average latency of the target device cluster in that round.
5. The method according to claim 1, characterized in that, If the target device cluster is a heterogeneous multi-device or heterogeneous 3D stacked device, the reference dataset also includes the computation time of each device for processing each expert, and the quantitative evaluation mechanism also considers the maximum computation latency of a single device in the cluster. An iterative optimization strategy involving pairwise expert exchanges between groups is employed to bring the final latency down to a preset range, resulting in the final expert storage allocation scheme. Specifically: The process iterates through the pairwise expert swap operations between groups, recalculating the latency of each swapped grouping scheme. If, after the swap, a high-activity-frequency expert migrates from a low-performance device to a high-performance device, causing the maximum latency to decrease and the average latency to not reach the set increase threshold, then the grouping scheme after the swap operation is used as the new expert storage allocation scheme. This process continues until the average latency of the cluster is within a preset range and the maximum computational latency of a single device is also within a set range. The iterative swap operation ends, and the expert storage allocation scheme after the last round of swap operations is used as the final expert storage allocation scheme.
6. The method according to claim 5, characterized in that, During the initial grouping, a load threshold is assigned to each device based on the differences in energy efficiency, so that high-performance devices can handle more experts than low-energy-efficiency devices.
7. The method according to claim 5, characterized in that, In the quantitative evaluation mechanism, the maximum computing latency of a single device is given a greater weight than the average latency of the cluster.
8. The method according to claim 5, characterized in that, The heterogeneous multi-devices include GPUs, CPUs, or dedicated acceleration chips with different architectures.
9. The method according to any one of claims 1-8, characterized in that, The reference dataset for constructing expert activation features forms expert activation frequency distribution data covering multiple preset task scenarios, specifically as follows: By statistically analyzing the expert activation records of the neural network layers of the hybrid expert model in the training set, test set, and various preset specific task scenarios, activation frequency distribution data covering multiple scenarios is formed.
10. A multi-device load balancing system with hybrid expert large language model reasoning, characterized in that, include: The dataset construction module is used to build a reference dataset of expert activation features, forming expert activation frequency distribution data covering multiple preset task scenarios; The initial grouping module is used to initially group all experts based on the number of target device clusters and the preset parallelism, thereby forming an initial storage allocation scheme. The initial latency calculation module is used to calculate the initial latency of the target device cluster using a quantitative evaluation mechanism for the initial grouping and reference dataset. The quantitative evaluation mechanism takes into account factors including the computational latency caused by the computation time of activating experts. The iterative optimization module is used to perform an iterative optimization strategy by exchanging experts in pairs between groups, so that the final delay reaches the preset range and the final expert storage allocation scheme is obtained.