Hybrid expert model inference joint deployment optimization method for heterogeneous GPU environment

Abstract

This invention discloses a joint deployment optimization method for hybrid expert model inference in heterogeneous GPU environments. The method first uses offline trajectory statistics to determine expert activation frequencies and collects resource profiles of heterogeneous devices. Then, it constructs a joint optimization model with the objective of minimizing the weighted sum of inference latency surrogate terms and quantization error penalty terms. Under memory constraints, a pruned greedy search algorithm is used to solve for the placement location, number, and quantization accuracy of each expert's replicas. During the online inference phase, a routing mechanism combining local priority and load-aware weighted round-robin is used for dynamic token distribution. This invention can simultaneously address inference latency, memory usage, and accuracy preservation in heterogeneous GPU environments, improving the processing efficiency of popular experts, reducing cross-device communication overhead and slow device trailing effects, and is suitable for distributed large-scale model inference deployment scenarios.

Description

Technical Field

[0001] This invention belongs to the field of large model inference optimization and heterogeneous computing resource scheduling technology, and in particular, it is a hybrid expert model inference joint deployment optimization method for heterogeneous GPU environments. Background Technology

[0002] With the widespread application of large language models in tasks such as intelligent dialogue, machine translation, and multimodal understanding and reasoning, how to efficiently deploy large language models in distributed computing environments has become a crucial issue in current artificial intelligence system research and engineering implementation. Compared to the unified and high-performance accelerator platforms configured in data centers, GPU devices in actual deployment environments typically exhibit significant heterogeneity, with substantial differences in computing power, memory capacity, and interconnect bandwidth between devices. Simultaneously, users' demands for low latency, high throughput, and stable service quality continue to increase, making efficient inference of large language models in heterogeneous GPU environments a technological direction with significant application value.

[0003] To reduce the enormous computational overhead of traditional dense large language models during the inference phase, hybrid expert model architectures are widely used in modern large language models. In the MoE framework, the gating network activates only a small number of experts for each input token, rather than activating all model parameters. This significantly reduces inference costs through sparse computation, making it possible to deploy ultra-large parameter-scale models on resource-constrained devices. Compared to traditional dense models, the MoE model can maintain strong model capabilities while more effectively adapting to resource-constrained and heterogeneous deployment scenarios through sparse activation, thus possessing high application value in edge inference and heterogeneous GPU environments.

[0004] Despite the significant advantages of the MoE model in terms of computational sparsity, its actual inference efficiency in heterogeneous GPU environments still faces severe challenges. These challenges mainly stem from the following aspects: 1) Uneven expert activation frequency. Different experts are selected at significantly different frequencies during operation, with a few popular experts carrying most of the routing traffic, while a large number of unpopular experts are accessed only infrequently. This significant popularity skew leads to load imbalance between devices and generates additional synchronization waiting overhead during expert parallel execution. 2) Significant device heterogeneity. Different GPUs differ in terms of computational throughput, memory budget, and communication bandwidth. If popular experts are deployed on weaker devices, it can easily create trailing devices and system bottlenecks, thereby increasing the overall inference latency. 3) Inter-coupling of multi-dimensional decisions. In the heterogeneous deployment scenario of MoE, the placement of experts and their replicas, the allocation method of expert quantization bit width, and the routing strategy from runtime tokens to expert replicas are not independent decisions, but rather mutually influential and restrictive. An unreasonable design in one dimension often simultaneously worsens load balancing, model accuracy, and end-to-end latency performance in other dimensions.

[0005] To improve the inference efficiency of large language models in heterogeneous GPU environments, existing research has proposed optimization strategies to address the aforementioned challenges. For example, Mei et al. (Y. Mei, Y. Zhuang, X. Miao, J. Yang, Z. Jia, and R. Vinayak, “Helix: Serving Large Language Models Over Heterogeneous GPUs and Network via Max-Flow,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages ​​and Operating Systems, Volume 1, 2025, pp. 586–602.) addressed the deployment problem of large language models in heterogeneous GPU environments by improving inference performance through a model placement optimization method for heterogeneous clusters. However, this method is mainly aimed at traditional dense large language models and does not support MoE model inference scenarios. Li et al. (J. Li, Y. Jiang, Y. Zhu, C. Wang, and H. Xu, “Accelerating Distributed MoE Training and Inference With Lina,” in 2023 USENIX Annual Technical Conference, 2023, pp. 945–959.) addressed the problem of uneven expert activation in the MoE model by balancing expert activation and replica scheduling mechanisms to alleviate the load pressure caused by popular experts. However, this method usually assumes that the underlying devices are homogeneous environments and fails to fully consider the differences in computing, memory, and communication capabilities of heterogeneous GPUs.Zhao et al. (J. Zhao, B. Wan, C. Wu, Y. Peng, and H. Lin, “Poster: LLM-PQ: Serving LLM on Heterogeneous Clusters With Phase-Aware Partition and Adaptive Quantization,” in Proceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, 2024, pp. 460–462.) optimized model deployment through adaptive quantization and heterogeneous cluster partitioning to reduce resource consumption and improve inference efficiency. However, their research mainly focused on traditional dense large language models, and did not adequately consider the differences in expert activation frequency and the issue of expert copy placement in MoE models. In addition, some studies have optimized the deployment of large models in heterogeneous environments from the perspectives of collaborative inference, parallelism on heterogeneous devices, or mixed-precision quantization, but most of these studies only address a single dimension and fail to systematically address the multi-dimensional coupling problem of MoE inference.

[0006] However, existing methods often have limitations: some methods ignore the significant difference in activation popularity between popular and unpopular experts in the MoE model when dealing with heterogeneous GPU deployments, leading to load imbalances even after expert placement. Other methods, while considering expert activation skew in the MoE model, fail to fully account for the complex differences in computing power, memory capacity, and inter-device bandwidth among heterogeneous GPUs, especially when popular experts are deployed to devices with weak computing power, which can easily create system bottlenecks. Furthermore, research on quantization often handles quantization separately from expert placement and runtime routing, failing to effectively integrate the coupling relationship between expert popularity, hardware heterogeneity, and real-time load status. As a result, even when utilizing the sparse activation characteristics of MoE, the system may still experience inference bottlenecks due to popular experts or replicas being assigned to weak devices, leading to decreased throughput, increased tail latency, and unstable performance.

[0007] In heterogeneous GPU deployments of the MoE model, expert placement, expert-level mixed-precision quantization, and runtime replica routing are naturally coupled. If expert placement and quantization are designed independently, the changes in parameter scale and computational cost brought about by quantization cannot be used to guide expert migration and load rebalancing. If unified quantization is performed without hardware heterogeneity awareness, popular experts may be deployed on devices with weaker computing power and limited precision support, further worsening the trade-off between throughput and accuracy. If the runtime routing strategy does not dynamically decide based on the real-time load of devices, local congestion and excessive cross-device communication overhead may still occur in multi-replica deployment scenarios.

[0008] In summary, a systematic solution is still lacking for MoE model inference in heterogeneous GPU environments, which suffers from low inference efficiency due to uneven expert activation frequency, device heterogeneity, and strong coupling between expert placement, quantization, and runtime routing. Therefore, designing a joint optimization method for MoE model inference that can adapt to heterogeneous hardware environments, significantly reduce end-to-end inference latency, improve overall throughput, and maintain model accuracy is crucial for promoting the practical application of large language models in edge and distributed heterogeneous environments. Summary of the Invention

[0009] The purpose of this invention is to address the problems existing in current hybrid expert model inference in heterogeneous GPU environments, such as concentrated load on popular experts, fragmented design of quantization and placement, high cross-device communication overhead, and lack of load awareness in multi-replica routing. This invention provides a joint optimization method and system for hybrid expert model inference in heterogeneous GPU environments. The method collaboratively optimizes expert replica placement and mixed-precision quantization in the offline phase, and introduces a replica routing mechanism that combines local priority and load-aware weighted round-robin in the online phase. This achieves joint and coordinated utilization of computing, storage, and communication resources in heterogeneous GPU environments, thereby reducing end-to-end inference latency, improving system throughput, and maintaining model accuracy.

[0010] The technical solution to achieve the purpose of this invention is: a hybrid expert model inference joint deployment optimization method for heterogeneous GPU environments, the method comprising the following steps:

[0011] Step 1: Construct a hybrid expert model distributed inference system on a heterogeneous GPU cluster, collect the memory budget, computing power and inter-device bandwidth information of each GPU device, and count the activation frequency of each expert at each layer based on the offline inference trajectory.

[0012] Step 2: Based on the activation frequency of experts, and with the goal of minimizing the weighted sum of the inference latency surrogate term and the quantization error penalty term, construct a joint optimization model of expert copy placement and mixed precision quantization that includes memory constraints.

[0013] Step 3: Based on the joint optimization model, solve for the replica placement configuration, quantization bit width and replica number of each expert on each GPU device to obtain the offline replica configuration result;

[0014] Step 4: During online inference, based on the offline replica configuration results, a replica routing mechanism combining local priority and load-aware weighted round-robin is used to complete token distribution and execution.

[0015] Furthermore, step 1 involves constructing a hybrid expert model distributed inference system on a heterogeneous GPU cluster, specifically including:

[0016] The hybrid expert model is deployed in a cluster consisting of multiple heterogeneous GPU devices, denoted as set. elements in Each GPU device has a corresponding video memory budget. ;

[0017] During the inference process, the shared sublayer processes the input token to obtain the token hidden state, and then the router selects the top-k target experts for each token;

[0018] The target experts are distributed across multiple GPU devices in a parallel manner, so that the target expert corresponding to the token can be located on both a local GPU device and a remote GPU device.

[0019] When the target experts are located on different GPU devices, the token hidden state is distributed through cross-device data exchange, and the expert output is sent back to the initiating device after the expert's calculation is completed.

[0020] Furthermore, step 1 involves calculating the activation frequency of each expert at each layer based on the offline inference trajectory, specifically including:

[0021] Acquire offline inference trajectory data and count the number of times each expert is activated in each layer;

[0022] Calculate the activation frequency of each expert within their respective layer;

[0023] The expert activation frequency is then expressed as:

[0024]

[0025] In the formula, Indicates the first Layered experts activation frequency, Indicates the first Layered experts Number of times it is activated Indicates the first A group of experts at the level Indicates the first Layer of experts The number of times it has been activated.

[0026] Furthermore, the joint optimization model of expert copy placement and mixed-precision quantization in step 2 is as follows:

[0027] For any expert Let the total number of its copies be . Then the following condition is met:

[0028]

[0029] And for the same expert The constraint for all replicas is to use a unique quantization bit width, so that the expert does not have multiple different quantization bit widths among all replicas;

[0030] in, Experts One Is the bit copy placed on the GPU device? superior; This represents a collection of GPU devices.

[0031] Furthermore, the objective function of the joint optimization model described in step 2 is expressed as follows: :

[0032]

[0033] In the formula, This indicates the placement and quantitative configuration of expert copies. To bring together all experts, For the set of candidate quantization bit widths, Indicates the inference delay surrogate term. This represents the quantization error penalty term. This represents the tradeoff coefficient between latency and quantization error;

[0034] The memory constraint is, for any GPU device The total memory usage of all expert copies and other model parameters deployed on this device does not exceed its video memory budget, expressed as:

[0035]

[0036] In the formula, Experts exist Parameter storage size under bit quantization Indicates equipment The model parameters and runtime overhead, excluding expert copies, are as follows. Indicates equipment The video memory budget.

[0037] Furthermore, the quantization error penalty term Represented as:

[0038]

[0039]

[0040] In the formula, Experts Achieving full precision Experts of Bit quantization implementation, Indicates routing to experts in offline tracks The set of hidden states Indicates configuration Next Layer experts The quantization bit width used, Experts Quantization error, Indicates the first A collection of experts at the level; Represents the square of the L2 norm. Indicates input to expert The hidden state.

[0041] Furthermore, the inference delay proxy item This includes computational delay proxy items and communication delay proxy items, represented as:

[0042]

[0043] To calculate the delay surrogate term, it is represented as:

[0044]

[0045] in,

[0046]

[0047]

[0048]

[0049] In the formula, Indicates deployment on GPU device Experts The ability to aggregate services Indicates deployment on GPU device Experts The ability to aggregate services Experts of Bit copies on GPU devices Service speed on; Experts Assigned to GPU devices The proportion of traffic, Experts Overall activation frequency, Indicates the first Layers on GPU devices The computational load on top, Indicates the first The average computational load of the layer across all GPU devices; This represents the communication latency weighting coefficient, used to balance the impact of unbalanced computational load and cross-device communication tailing on overall latency. Indicates the first A collection of experts at the level;

[0050] For communication delay proxy items, it is represented as:

[0051]

[0052] In the formula, Indicates in configuration The following equipment Issued and in the device The expected rate of token processing is as follows: Indicates GPU device With GPU devices The effective bandwidth between them.

[0053] Furthermore, in step 3, a pruning greedy search method is used to solve for the replica placement configuration of each expert on each GPU device, quantizing the bit width and the number of replicas. The specific process includes:

[0054] Step 3-1: Divide all experts into groups to obtain the expert group set. Experts within the same expert group share the quantization bit width. To reduce the quantization search space;

[0055] Step 3-2: Sort experts by activation frequency from high to low, and assign copies to experts accordingly. In each assignment, select the proxy that can reduce inference latency. The smallest GPU device is the target deployment device;

[0056] Step 3-3: After obtaining the initial deployment results, identify the tailing device set and the error-dominant expert set, and perform a local search, focusing only on the top experts in the tailing device set. The top GPU devices and error-dominant expert set Several experts conduct copy migration or bit width reduction searches; based on the current configuration, the computational and communication loads of all devices are analyzed, and a set of devices whose contribution to overall latency is greater than a first preset threshold is selected to form a tailing device set. Based on the contribution of the quantified error penalty term, experts whose impact on overall accuracy is greater than a preset second threshold are selected to form an error-dominant expert set. ;

[0057] Steps 3-4: When a candidate move causes a change in the objective function... When the candidate move is accepted, the current configuration is updated, and the local search continues until no candidate move is found that causes the objective function to continue to decrease.

[0058] Furthermore, the candidate operations for the local search in step 3-3 include at least:

[0059] Remove expert copy from GPU device Migrate to GPU devices The copy migration operation; and,

[0060] Quantization bit width of the current expert group Reduce to a lower candidate bit width And satisfy Bit width adjustment operation;

[0061] During each candidate operation evaluation, the objective function change and memory constraints are simultaneously checked.

[0062] Furthermore, the replica routing mechanism combining local priority and load-aware weighted round-robin described in step 4 satisfies the following: when a local replica of the target expert exists, local execution is always given priority; when remote execution is the only option, tokens are preferentially distributed to GPU devices with a service rate greater than the first preset threshold and a prediction queue length less than the second preset threshold, so as to achieve cross-device load balancing while maintaining the local priority principle; wherein the first preset threshold and the second preset threshold are customizable.

[0063] Furthermore, the replica routing mechanism combining local priority and load-aware weighted round-robin specifically includes:

[0064] For each input token, first determine whether there is a local copy of the target expert on the current GPU device. If there is a local copy, prioritize routing to the local copy for execution. If there is no local copy, or the local execution load exceeds the preset conditions, perform load estimation based on the queue length and service rate of the candidate remote copies, and use a load-aware weighted round-robin strategy to select the target remote copy.

[0065] Furthermore, the replica routing mechanism combining local priority and load-aware weighted round-robin specifically includes:

[0066] If the GPU device currently running the computation has a target expert If the local copy is used, the current token will be directly routed to the local copy for execution;

[0067] If the GPU device currently running the computation does not have a target expert If the local copy is not found, or if the local copy's execution load exceeds a preset condition, the current token will be distributed to the target expert. One of the candidate remote replicas in the set;

[0068] For each candidate remote copy's GPU device ,set up Representing discrete time expert In GPU devices The number of unprocessed tokens corresponding to the above replica. This indicates the service rate of the replica within the window length. Internal estimation of next moment expert Reaching GPU devices Recent arrival rate :

[0069]

[0070] In the formula, This indicates the time t for the GPU device. Top experts The estimated token arrival rate.

[0071] Dynamic routing weights are constructed based on the predicted queue length and service rate:

[0072]

[0073] in, To prevent positive numbers with a denominator of zero.

[0074] The target remote GPU device is selected from the candidate remote GPU devices using a weighted polling function. :

[0075]

[0076] And route the current token to the expert copy on the selected GPU device for execution;

[0077] In the formula, This represents the weighted polling function. Experts The GPU device it is located on.

[0078] Compared with the prior art, the significant advantages of this invention are:

[0079] (1) Incorporate both expert activation frequency and GPU heterogeneity into deployment decisions. This invention fully considers the significant difference in popularity between popular experts (i.e., high activation frequency experts) and unpopular experts (i.e., low activation frequency experts) in the MoE model, and combines the differences in computing, memory and bandwidth capabilities of heterogeneous GPUs for unified modeling, which can avoid the problem of device utilization imbalance caused by simply allocating according to average or single popularity strategy.

[0080] (2) Joint optimization of expert copy placement and quantization accuracy. This invention no longer designs quantization and placement separately, but coordinates the quantized parameter volume, expert service rate and equipment resource capabilities to enable popular experts to obtain more reasonable deployment positions and accuracy configurations, thereby reducing memory usage and computation latency while controlling accuracy loss.

[0081] (3) Introducing a local priority and load-aware weighted round-robin routing mechanism. This invention prioritizes the execution of tokens on local replicas, and only dynamically selects among multiple candidate replicas when remote execution is necessary. It also prioritizes the distribution of tokens to high-capacity, low-congestion target replicas, thereby reducing cross-device communication overhead and queue waiting latency.

[0082] (4) Improve the overall efficiency and stability of MoE inference in a heterogeneous GPU environment. This invention achieves unified coordination of computing, storage and communication resources by combining offline joint optimization and online dynamic routing. It can significantly improve the system throughput of MoE model inference, reduce the latency of the first token and the latency of decoding each token in a complex heterogeneous environment, and ensure the stability and scalability of the overall operation.

[0083] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0084] Figure 1 This is a flowchart illustrating the joint deployment optimization method for hybrid expert model inference in heterogeneous GPU environments according to the present invention. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0086] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0087] The purpose of this invention is to address the problems existing in existing hybrid expert model inference in heterogeneous GPU environments, such as concentrated load of popular experts, fragmented design of quantization and placement, high cross-device communication overhead, and lack of load awareness in multi-replica routing, and to provide a joint optimization method and system for hybrid expert model inference in heterogeneous GPU environments.

[0088] In one embodiment, combined Figure 1 This paper provides a joint optimization method for hybrid expert model inference in heterogeneous GPU environments, the method comprising:

[0089] Step 1: Establish a distributed inference system model of hybrid expert model in heterogeneous GPU environment, and count the activation frequency of each expert in each layer according to the offline inference trajectory. At the same time, collect the memory budget, computing power, service rate under different quantization precision and effective bandwidth information between devices of each GPU device.

[0090] Step 2: Construct a joint expert replica placement and mixed-precision quantization optimization model. Specifically, quantization error penalty, computational load imbalance, communication tail cost, and memory constraints are all incorporated into a unified objective function to jointly model the deployment location of expert replicas on heterogeneous GPUs and the quantization bit width.

[0091] Step 3: Use a pruning greedy search method to solve the offline replica configuration. Specifically, first, initialize expert precision groups based on expert activation frequency, then complete the greedy placement of replicas in descending order of expert frequency. Subsequently, perform a local search for trailing devices and high-error dominant experts to obtain replica configuration results that satisfy memory constraints while balancing latency and accuracy.

[0092] Step 4: During the online inference phase, runtime replica routing is performed based on a local priority and load-aware weighted round-robin mechanism. For each input token, if a replica of the target expert exists on the local device, it is executed locally first. If no target replica exists locally, or if it is necessary to select from multiple remote replicas, the routing weight is dynamically calculated based on the recent arrival rate, queue length, and service rate of the candidate replicas, and the target remote replica is selected through weighted round-robin.

[0093] Step 5: Based on the offline configuration and online routing results, complete token distribution, expert execution, expert output feedback, and subsequent layer inference, thereby achieving high throughput and low latency inference of the hybrid expert model in a heterogeneous GPU environment.

[0094] Furthermore, in one embodiment, step 1 specifically includes:

[0095] Step 1-1: Establish a distributed inference system model for a hybrid expert model on a heterogeneous GPU cluster. Let the set of heterogeneous GPU devices be denoted as . Any of the GPU devices is denoted as Its corresponding video memory budget is Let the set of all experts be... The candidate quantization bit width set is During distributed inference, each GPU device replicates the shared sublayer, and expert modules are distributed across multiple devices in an expert-parallel manner.

[0096] Steps 1-2 establish the token processing flow model for MoE inference. Each input token first passes through a shared sublayer to obtain a hidden state, and then a gating network selects the top-k target experts for that token. Since the target experts may be located on different GPU devices, the system sends the token's hidden state to the device where the corresponding expert's copy is located through cross-device data exchange, and after the expert's calculation is completed, the expert's output is sent back to the original device for aggregation.

[0097] Steps 1-3: Statistical expert activation frequency. Based on offline inference trajectories, calculate the... Layered experts activation frequency The activation frequency is expressed as:

[0098]

[0099] in, Indicates the first Layered experts The number of times it was activated in the offline trajectory. Indicates the first A group of experts at the level. This is used to characterize the relative frequency of experts; popular experts correspond to higher activation frequencies, while less popular experts correspond to lower activation frequencies.

[0100] Steps 1-4: Collect information from heterogeneous devices. For each GPU device, collect its memory budget, expert service rate at different quantization bit widths, device computing power, and effective bandwidth of inter-device links for subsequent joint optimization modeling.

[0101] Furthermore, in one embodiment, step 2 specifically includes:

[0102] Step 2-1: Establish expert copy placement variables and quantification variables. Define the decision variables. Experts One Is the bit copy placed on the GPU device? This constitutes a set of placement and quantitative configuration. .in, Indicates expert index, Indicates the GPU device index. This indicates the candidate quantization bit width. For the same expert, all copies use the same quantization precision to avoid excessive precision inconsistencies between different copies of the same expert.

[0103] Step 2-2: Establish the joint optimization objective function. The inference delay surrogate term and the quantization error penalty term are jointly incorporated into the objective function, expressed as:

[0104]

[0105] in, This represents the overall target value under configuration Z. Indicates the inference delay surrogate term. This represents the quantization error penalty term. This represents the trade-off coefficient between latency and quantization error. By minimizing the objective function, the end-to-end inference latency of the system is reduced while controlling for accuracy loss.

[0106] Steps 2-3: Establish the quantization error penalty model. For the first... Layer experts and candidate quantization bit width ,set up Experts Achieving full precision Indicate its Bit quantization implementation, This indicates that the hidden state set of the expert is routed to in the offline trajectory, and therefore the expert has high quantization accuracy. The quantization error is expressed as follows:

[0107]

[0108] in, Indicates input to expert The hidden state, This represents the squared L2 norm. This formula is used to quantify the degree of distortion in expert output between full precision and quantized precision.

[0109] Furthermore, the totalization error penalty term is expressed as:

[0110]

[0111] in, Indicates in configuration Next Layer experts The quantization bit width used, This is used to reflect the weighting effect of expert popularity on quantization error. Popular experts have a greater impact on the overall model output, therefore their quantization errors have a higher weight in the optimization process.

[0112] Steps 2-4: Establish a computational load balancing agent model. Assume an expert... In GPU devices The aggregation service capabilities are as follows:

[0113]

[0114] in, Experts In GPU devices above The service rate at bit-precision operation. A higher service rate indicates a stronger ability of the device to handle expert requests.

[0115] Under the capacity-ratio routing approximation, experts Assigned to GPU device The traffic ratio is expressed as:

[0116]

[0117] in, Experts Overall activation frequency, Represents the index of any device in the GPU device set U.

[0118] Furthermore, the first Layers on GPU devices The computational load on it is represented as:

[0119]

[0120] in, Indicates the first All experts at the level are on the equipment The total computing load on the device.

[0121] Furthermore, the load imbalance term is expressed as follows:

[0122]

[0123] in, Indicates the first The average computational load of a layer across all GPU devices. This formula is used to explicitly characterize the degree of load imbalance between devices, thereby suppressing the formation of tailing devices.

[0124] Steps 2-5: Establish the communication tail proxy model. Let... Indicates in configuration The following is from the GPU device Issued and on GPU device The expected rate of tokens processed by experts. Indicates equipment With equipment The effective bandwidth between them is then represented by the communication tail term as follows:

[0125]

[0126] in, This formula is used to characterize the dominant impact of the slowest link on overall communication latency. It can reflect the adverse effects of low-bandwidth links or concentrated cross-device traffic on system inference performance.

[0127] Furthermore, the overall inference delay surrogate term is represented as:

[0128]

[0129] in, This represents the communication latency weighting coefficient, used to balance the impact of unbalanced computational load and cross-device communication tailing on overall latency.

[0130] Steps 2-6: Establish the memory constraint model. For any GPU device... The total memory usage of all expert copies and other model parameters deployed on this device does not exceed its video memory budget, expressed as:

[0131]

[0132] in, Experts exist Parameter storage size under bit quantization Indicates equipment The model parameters and runtime overhead, excluding expert copies, are as follows. Indicates equipment The video memory budget.

[0133] Furthermore, in one embodiment, step 3 specifically includes:

[0134] Step 3-1: Sort all experts according to their activation frequency, and initialize the precision groups of experts according to their frequency. This will allow popular experts to be assigned higher quantization precision, while less popular experts will be assigned a candidate set with lower quantization precision, thereby reducing the search space.

[0135] Step 3-2: A greedy construction method is used to sequentially select the deployment location and quantization precision for each expert replica. For each replica to be placed, from the candidate device-precision pairs that satisfy the memory constraints, the candidate that minimizes the increment of the objective function is selected as the current optimal deployment result.

[0136] Step 3-3: Identify the tailing device set and the error-dominant expert set, and perform a local search on them. The local search includes: migrating a copy of an expert from the current device to another candidate device, or adjusting the quantization bit width of a group of experts to a lower candidate bit width.

[0137] Steps 3-4: If a local shift causes a change in the objective function... If a local move is found that can further reduce the objective function, the search is terminated and the offline replica configuration result is output.

[0138] Furthermore, in one embodiment, step 4 specifically includes:

[0139] Step 4-1 defines the runtime replica routing strategy as a local priority strategy. For any input token, if its target expert has a local replica on the GPU device where the current computation is located, the local replica will always be selected for execution to reduce cross-device communication overhead.

[0140] Step 4-2: When the target expert does not have a copy on the local device, or when a selection needs to be made among multiple remote copies, perform load estimation on each candidate remote copy. Let... Representing discrete time expert In GPU devices The number of unprocessed tokens corresponding to the above replica. This indicates the service rate of the replica, which is within the window length. The recent arrival rate within the area is expressed as:

[0141]

[0142] in, This formula represents the predicted queue length for the next time step. It is used to simultaneously consider the current queuing status, recent arrival trends, and service capacity. This indicates the time t for the GPU device. Top experts The estimated token arrival rate.

[0143] Step 4-3: Calculate the dynamic routing weight based on the predicted queue length and service rate, expressed as:

[0144]

[0145] in, Indicates time expert In the equipment Dynamic routing weights for the replica To prevent positive integers with a denominator of zero, the higher the service rate and the shorter the prediction queue, the greater the corresponding weight.

[0146] Step 4-4: Based on the dynamic routing weights, a weighted round-robin function is used to select the target replica from the candidate remote devices, expressed as:

[0147]

[0148] in, Indicates the selected target remote GPU device. This represents a weighted round-robin function. This method achieves dynamic load balancing of remote replicas while maintaining the local priority principle.

[0149] Furthermore, in one embodiment, step 5 specifically includes:

[0150] Step 5-1: Based on the offline replica configuration results obtained in Step 3, deploy the experts and their replicas in the hybrid expert model to the corresponding heterogeneous GPU devices, and load the expert parameters of the corresponding precision according to the quantization configuration.

[0151] Step 5-2: During the inference process, the shared sublayer generates a hidden token state and determines the target expert copy corresponding to each token according to step 4.

[0152] Step 5-3: Distribute the hidden state of the token to the device where the target expert copy is located, complete the expert calculation, and then send the output back to the original device for aggregation.

[0153] Step 5-4 repeats the shared computation, expert routing, cross-device communication, and output aggregation of subsequent layers until the entire inference process is completed.

[0154] In one embodiment, a high-throughput inference system for hybrid expert models in heterogeneous GPU environments is provided, the system comprising:

[0155] The first module (profile analysis module) is used to count the activation frequency of experts in offline inference trajectories and to collect information on the memory budget, computing power, quantization service rate, and inter-device bandwidth of each GPU device.

[0156] The second module (Joint Placement and Quantization Programming Module) is used to construct a joint optimization model that simultaneously includes quantization error penalties, computational load imbalance, communication tail costs, and memory constraints, and to solve for expert copy placement schemes and quantization schemes.

[0157] The third module (replica routing module) is used to select the target expert replica at runtime according to the local priority and load-aware weighted round-robin mechanism, so as to realize dynamic routing and load balancing of tokens.

[0158] The fourth module (inference execution module) is used to complete token distribution, expert execution, output return, and subsequent layer inference based on offline copy configuration and online routing results.

[0159] Specific limitations regarding the high-throughput inference system for hybrid expert models in heterogeneous GPU environments can be found in the limitations of the joint deployment optimization method for hybrid expert model inference in heterogeneous GPU environments described above, and will not be repeated here. Each module in the aforementioned high-throughput inference system for hybrid expert models in heterogeneous GPU environments can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0160] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0161] Step 1: Construct a hybrid expert model distributed inference system on a heterogeneous GPU cluster, collect the memory budget, computing power and inter-device bandwidth information of each GPU device, and count the activation frequency of each expert at each layer based on the offline inference trajectory.

[0162] Step 2: Based on the activation frequency of experts, and with the goal of minimizing the weighted sum of the inference latency surrogate term and the quantization error penalty term, construct a joint optimization model of expert copy placement and mixed precision quantization that includes memory constraints.

[0163] Step 3: Based on the joint optimization model, solve for the replica placement configuration, quantization bit width and replica number of each expert on each GPU device to obtain the offline replica configuration result;

[0164] Step 4: During online inference, based on the offline replica configuration results, a replica routing mechanism combining local priority and load-aware weighted round-robin is used to complete token distribution and execution.

[0165] For specific limitations on each step, please refer to the limitations on the joint deployment optimization method for hybrid expert model inference in heterogeneous GPU environments mentioned above, which will not be repeated here.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0167] Step 1: Construct a hybrid expert model distributed inference system on a heterogeneous GPU cluster, collect the memory budget, computing power and inter-device bandwidth information of each GPU device, and count the activation frequency of each expert at each layer based on the offline inference trajectory.

[0168] Step 2: Based on the activation frequency of experts, and with the goal of minimizing the weighted sum of the inference latency surrogate term and the quantization error penalty term, construct a joint optimization model of expert copy placement and mixed precision quantization that includes memory constraints.

[0169] Step 3: Based on the joint optimization model, solve for the replica placement configuration, quantization bit width and replica number of each expert on each GPU device to obtain the offline replica configuration result;

[0170] Step 4: During online inference, based on the offline replica configuration results, a replica routing mechanism combining local priority and load-aware weighted round-robin is used to complete token distribution and execution.

[0171] For specific limitations on each step, please refer to the limitations on the joint deployment optimization method for hybrid expert model inference in heterogeneous GPU environments mentioned above, which will not be repeated here.

[0172] As a specific example, the invention will be described in detail in some embodiments.

[0173] This embodiment provides a joint optimization method for hybrid expert model inference in heterogeneous GPU environments, the method comprising the following steps:

[0174] Step 1: Establish a distributed inference system model of hybrid expert model in heterogeneous GPU environment, and count the activation frequency of each expert in each layer according to the offline inference trajectory. At the same time, collect the memory budget, computing power, quantization service rate and effective bandwidth information between each GPU device.

[0175] The process involves establishing a hybrid expert model distributed inference system model in a heterogeneous GPU environment and statistically analyzing expert activation frequencies, as detailed below:

[0176] Step 1-1: Establish a heterogeneous GPU cluster model. Let the set of heterogeneous GPU devices be... ,in Represents any GPU device. This represents the total number of devices. Each GPU device corresponds to a specific memory budget. Device computing power Effective bandwidth between devices and expert service rates under different quantization bit widths .

[0177] Steps 1-2 establish the distributed inference process of the hybrid expert model. For the input token, the shared sublayer first generates a hidden token state, and then the gating network selects the top-k target experts. If the target expert is located on the local device, it is executed directly locally. If the target expert is located on a remote device, the hidden token state is sent to the device where the target expert's copy is located via cross-device exchange. After the expert completes the calculation, the expert output is returned to the original device for aggregation and sent to the next shared sublayer. Therefore, the system performance is not only related to the calculation time of a single expert, but also closely related to the expert's placement location, link bandwidth, and cross-device communication mode.

[0178] Steps 1-3: Establish the expert set and the candidate accuracy set. Let the total expert set be... The candidate quantization bit width set is .in, This represents the set of all expert indexes in the hybrid expert model. This indicates the set of allowed quantization bit widths. In one specific embodiment, popular experts are given priority for higher precision, while less popular experts are allowed to use lower precision, in order to reduce memory usage and computational overhead without significantly reducing the overall precision.

[0179] Steps 1-4: Establish an offline trajectory statistical model. Based on a representative offline dataset D, statistically analyze the number of times each level of experts is routed. Let the... Layered expert group , No. Layered experts The number of times it was activated was Then the experts in that layer activation frequency It can be represented as:

[0180]

[0181] In the formula, Indicates the first Layered experts The relative activation frequency. Experts The number of activations in the offline inference trajectory. Indicates the first The sum of the number of times all experts in the layer are activated. Experts can be categorized into popular and unpopular experts based on this activation frequency, allowing for differentiated handling during subsequent placement and quantification.

[0182] Steps 1-5: Establish a device performance profile model. For any device... Given any candidate accuracy, the average service rate of experts on this device is obtained through offline profiling. To facilitate subsequent optimization, define... Indicates equipment With equipment The effective bandwidth between them. The portrait analysis results together constitute the input of the joint optimization model.

[0183] Steps 1-6 establish a cross-device traffic description for the token within the system. For any two devices... and If a target expert copy of a certain token is deployed on the device The shared sublayer of this token is computed on the device. If completed above, the token needs to be routed along the device. To the equipment The link performs a forward distribution, and after the expert output is formed, it is distributed again along the device. To the equipment The link return. Therefore, the deployment location of popular experts on different devices directly affects the distribution of cross-device traffic and further impacts system communication latency.

[0184] Steps 1-7 allow the profiling analysis module to be deployed as an offline data collection unit within the coordination node. This unit aggregates expert popularity statistics, video memory usage, and device performance information under representative workloads. The coordination node then transmits the profiling results to the subsequent joint placement and quantitative planning module, as well as the runtime replica routing module, to ensure consistency between offline structural decisions and online execution control.

[0185] Step 2: Construct a joint expert replica placement and mixed-precision quantization optimization model. Specifically, quantization error penalty, computational load imbalance, communication tail cost, and memory constraints are incorporated into a unified objective function to jointly model expert replica placement and mixed-precision quantization.

[0186] The construction of the joint expert replica placement and mixed precision quantization optimization model is as follows:

[0187] Step 2-1: Establish placement and quantification variables. Define a binary decision variable. Experts One Is the bit replica deployed on the device? superior. like This indicates that the expert In the equipment A precision of 1000 was deployed on it. A copy. If If not, it means it is not deployed. Therefore, the overall configuration set... It can be represented as:

[0188]

[0189] To control accuracy consistency, in one embodiment, all copies of the same expert are constrained to use a uniform quantization bit width.

[0190] Step 2-2: Establish a joint optimization objective function. To control model accuracy loss while reducing system inference latency, this invention incorporates both the inference latency surrogate term and the quantization error penalty term into a unified objective function, expressed as:

[0191]

[0192] In the formula, Indicates in configuration The overall target value is as follows: Indicates the inference delay surrogate term. This represents the quantization error penalty term. This represents the trade-off coefficient between latency and quantization error. By minimizing this objective function, a balance can be achieved between latency and accuracy.

[0193] Steps 2-3: Establish an expert-quantified error model. Let... Indicates the first Layer experts Achieving full precision The expert stated that... Quantization implementation at bit precision Let represent the set of hidden states in the offline trajectory that are routed to the expert. Then, the quantization error can be expressed as:

[0194]

[0195] In the formula, Indicates the first Layer experts exist Quantization error at bit precision Represents a set The number of samples, Indicates input to expert The hidden state, This represents the squared L2 norm. This formula is used to describe the average degree of deviation in expert output before and after quantization.

[0196] Steps 2-4: Construct the overall quantization error penalty term. Since popular experts have a greater impact on the final model output, expert popularity weighting is introduced into the quantization error, resulting in the overall quantization error penalty term:

[0197]

[0198] In the formula, Indicates configuration Overall quantization error penalty Indicates configuration Next Layer experts The quantization bit width used, Indicates the first Layer experts The activation frequency. This popularity-weighted approach allows for more careful protection of the quantitative accuracy of popular experts.

[0199] Steps 2-5: Establish an expert aggregation service capability model. For any expert... and equipment In configuration Its aggregated service capabilities can be represented as follows:

[0200]

[0201] In the formula, Indicates equipment Above and experts Related overall service capabilities Experts In the equipment above The service rate at bit-precision runtime. If no corresponding replica is deployed at this precision, the corresponding item is zero.

[0202] Steps 2-6: Establish a capacity-ratio traffic allocation model. Under the capacity-ratio routing approximation, experts... Distributed to equipment Traffic ratio It can be represented as:

[0203]

[0204] In the formula, Experts Overall activation frequency, This represents the expert's total service capacity across all devices. This formula reflects how the actual load distribution is jointly determined by the expert's popularity and the device's processing capacity.

[0205] Steps 2-7: Establish a hierarchical load calculation model. Let the first... Layer in device The computational load on it is:

[0206]

[0207] In the formula, Indicates the first All experts at the level are on the equipment The cumulative computational load on the device.

[0208] Furthermore, a computational load imbalance term is established:

[0209]

[0210] In the formula, Indicates configuration The following calculates the load imbalance item. Indicates the first The average computational load of the layer across all devices. The larger this value is, the more unbalanced the load is between different devices, and the more likely it is to create trailing devices.

[0211] Steps 2-8: Establish the communication tailing cost model. Let... Indicates in configuration The following equipment Issued and in the device The token traffic rate that has been processed by experts Indicates equipment With equipment Given the effective bandwidth between them, the communication tail term can be expressed as:

[0212]

[0213] In the formula, This represents the slowest communication item in the entire link. Indicates configuration Down from the device To the equipment Token traffic strength This represents the effective bandwidth of the corresponding link. This formula is used to reflect the dominant impact of low-bandwidth links and high cross-device traffic concentration on end-to-end latency.

[0214] Steps 2-9: Establish the overall latency proxy item. Considering the combined effects of load imbalance and communication tailing, define:

[0215]

[0216] In the formula, This represents the overall reasoning delay surrogate term. This represents the weighting coefficient of the communication tail term. It can be adjusted... The size of the tail can control the relative importance of the computational tail and the communication tail in the objective function.

[0217] Step 2-10, establish the memory constraint model. Let... Experts exist Parameter storage size at bit precision Indicates equipment Excluding the expert copy and other model parameters and runtime overhead, the memory constraint can be expressed as:

[0218]

[0219] In the formula, Indicates equipment The video memory budget, This represents the video memory used by shared sublayers, caches, and runtime overhead. The constraint ensures that the deployment of copies on any device does not exceed its video memory capacity.

[0220] The joint optimization model constructed in step 2 does not solve replica placement and quantization accuracy separately, but rather considers the mutual influence of the two types of decisions simultaneously through a unified objective function. Since quantization accuracy changes the storage volume and service rate of experts, while replica placement changes cross-device traffic distribution and load balancing, only by modeling both in a unified manner can we avoid the problem of popular experts being deployed to weak devices, or the problem of popular experts having excessively large output errors due to low-precision quantization.

[0221] Step 3: Use a pruning greedy search method to solve the offline replica configuration. First, initialize the precision grouping based on the expert activation frequency. Then, complete the greedy placement of replicas according to the expert activation frequency from high to low. Subsequently, perform a local search for trailing devices and error-dominant experts to obtain the replica configuration result that satisfies the memory constraints and takes into account both latency and accuracy.

[0222] The offline replica configuration is solved using a pruning greedy search method, as detailed below:

[0223] Step 3-1: Initialize precision grouping. Based on the expert popularity statistics obtained in Step 1, divide all experts into several groups according to their popularity distribution, and initialize different quantization precisions for different popularity groups. Generally, popular experts are assigned higher precision, moderately popular experts are assigned medium precision, and less popular experts are allowed to use lower precision. This method can significantly reduce the search space and lower the probability of serious precision loss in the initial deployment phase.

[0224] Step 3-2: Sort experts by popularity from highest to lowest, and process each expert's copy in turn. Assume... This represents the set of candidate device-precision pairs for the current expert copy.

[0225] Step 3-3, in the candidate set Select the device-precision pair with the lowest incremental cost as the deployment result for the current replica:

[0226]

[0227] In the formula, Indicates that experts The One copy is deployed to the device. And adopt precision The resulting increment in the objective function, This indicates the optimal candidate deployment result. This step is repeated until all expert replicas have obtained the initial placement result.

[0228] Steps 3-4: Identify the set of trailing devices Based on the current initial configuration, the computational and communication loads of all devices are analyzed, and several devices that contribute significantly to the overall latency are selected to form a tailing device set. This is used for subsequent local searches. A "tailing device" refers to a device that significantly increases the overall latency under the current configuration due to excessive load or limited link capacity.

[0229] Steps 3-5: Identify the expert set that dominates the error. Based on the contribution of the quantified error penalty term in Step 2, select the expert set that has a significant impact on the overall accuracy. Generally, if an expert has both high popularity and large quantification error, then that expert is given priority in the local search candidate.

[0230] Steps 3-6 involve performing a partial search. A partial search includes two types of operations: the first is a replica migration operation, which involves migrating a specific replica from the device... Migrate to device The second category is precision adjustment operations, which involve increasing or decreasing the quantization bit width of a particular expert group. If a candidate operation causes the objective function to change by a certain amount, then... If the objective function is not found, the operation is accepted and the current configuration is updated. Otherwise, the operation is discarded. The local search continues until no further candidate updates that reduce the objective function are found.

[0231] Steps 3-7, in one embodiment, to reduce search complexity, the local search does not perform an exhaustive search on all devices and all experts, but only on the tailing device set. Error-dominant expert set This allows for execution on the ground, thereby reducing offline solution time while ensuring solution quality.

[0232] Steps 3-8 output the offline replica configuration results. These results should include at least: the deployment devices for each expert replica, the quantization bit width used by each expert, the expert distribution on each device, and the replica quantity configuration. This result will serve as the static input for the online replica routing mechanism.

[0233] Step 4: During the online inference phase, a routing mechanism that prioritizes local execution and load-aware weighted round-robin is adopted for each input token to achieve dynamic load balancing among remote replicas while maintaining local execution priority.

[0234] The runtime replica routing based on the local priority and load-aware weighted round-robin mechanism is performed as follows:

[0235] Step 4-1: Establish a local priority routing rule. For any input token, if its target expert has a local copy on the current computing device, then that local copy will always be selected for execution first. This rule minimizes cross-device communication overhead and is a fundamental principle of runtime copy routing.

[0236] Step 4-2: When the target expert does not have a copy on the local device, or when a selection needs to be made among multiple remote copies, perform load estimation on each candidate remote copy. Let... Experts representing discrete time t In GPU devices The number of unprocessed tokens corresponding to the above replica. This indicates the service rate of the replica, which is within the window length. The recent arrival rate within the area is expressed as:

[0237]

[0238] in, This formula represents the predicted queue length at the next moment. It is used to simultaneously consider the current queuing status, recent arrival trends, and service capacity.

[0239] Step 4-3: Calculate the dynamic routing weight based on the predicted queue length and service rate, expressed as:

[0240]

[0241] in, Indicates time expert In the equipment Dynamic routing weights for the replica To prevent positive integers with a denominator of zero, the higher the service rate and the shorter the prediction queue, the greater the corresponding weight.

[0242] Step 4-4: Based on the dynamic routing weights, a weighted round-robin function is used to select the target replica from the candidate remote devices, expressed as:

[0243]

[0244] in, Indicates the selected target remote GPU device. This represents a weighted round-robin function. This method achieves dynamic load balancing of remote replicas while maintaining the local priority principle.

[0245] The routing mechanism described in step 4 combines determinism and dynamism: when a local replica exists, a deterministic local priority strategy is always executed. When only remote replication is possible, dynamic selection is performed based on real-time load. Therefore, this mechanism reduces unnecessary cross-device transmissions and avoids local congestion between remote replicas.

[0246] Step 5: Based on the offline replica configuration and online routing results, complete token distribution, expert execution, output feedback, and subsequent layer inference, thereby achieving high throughput and low latency inference of the hybrid expert model in a heterogeneous GPU environment.

[0247] The process of token distribution, expert execution, and output feedback based on offline configuration and online routing results is as follows:

[0248] Step 5-1: Based on the offline replica configuration results obtained in Step 3, deploy the experts and their replicas in the hybrid expert model to the corresponding heterogeneous GPU devices, and load the corresponding parameters according to the quantization accuracy obtained from the solution.

[0249] In step 5-2, during the inference process, the shared sublayer performs common computations to generate the token's hidden state, and the gating network determines the target expert set required for that token. For a token that requires processing by multiple experts, the expert outputs are ultimately aggregated through a weighted method.

[0250] Step 5-3, set the gating network as the target expert. The generated normalized weights are The output of this token in the current MoE layer can be represented as:

[0251]

[0252] In the formula, This indicates the final output of the token in the current MoE layer. Indicates that the gating network is the input. The selected top-k expert set Experts The corresponding gating weights, Experts For input The output of this expression indicates that the execution results of multiple expert copies are ultimately used to form the output of this layer in a gated weighted sum manner.

[0253] Step 5-4: Based on the routing result obtained in step 4, send the hidden state of the token to the device where the target expert copy is located. If the local copy is matched, the operation is performed locally. If remote operation is required, the hidden state of the token is sent across devices and the expert calculation is completed on the target device.

[0254] Step 5-5: After the target expert copy completes the computation, the expert output is sent back to the original device, weighted and aggregated according to the gating weights, and then sent to the next shared sublayer. This process of shared computation, expert routing, expert execution, and output aggregation is repeated until the computation of all inference layers is completed.

[0255] The system's operational and offline phases form a closed loop: the offline phase generates replica configurations through profiling and joint planning, the online phase executes routing through local priority and load awareness mechanisms, and the statistical information accumulated in the online phase can be fed back for the next offline planning update, thereby enabling the system to gradually adapt to changes in workload and device status.

[0256] Furthermore, in a more specific embodiment, to facilitate the subsequent joint planning, the expert popularity described in step 1 can be graded. Let... , Then the first Layered experts Normalized heat It can be represented as:

[0257]

[0258] In the formula, Indicates the first Layered experts Normalized heat To prevent positive numbers with a denominator of zero. According to The value of this parameter can be used to categorize experts into popular and unpopular expert groups. This allows for tiered control of subsequent quantization initialization and replica allocation processes based directly on the normalized popularity.

[0259] Furthermore, in a more specific embodiment, a popularity threshold can be preset. and the threshold for unpopularity ,in .when At that time, experts Determined to be a popular expert. When At that time, experts These are identified as niche experts. Using the methods described above, different copy configuration and quantification strategies can be applied to experts at different popularity levels.

[0260] The device performance profiling described in step 1 also includes dynamic correction of the bandwidth matrix and service rate matrix. That is, during actual system operation, the actual effective bandwidth of the links between devices and the average processing time of expert replicas can be recorded periodically, and the initial profiling results can be updated to improve the adaptability of subsequent offline planning to the real operating environment.

[0261] In step 2, in addition to memory constraints, constraints are also placed on the number of expert replicas and the uniqueness of a single device. Let... Experts The total number of copies is:

[0262]

[0263] For any device, the same expert At most one replica can be deployed on this device, resulting in:

[0264]

[0265] This constraint ensures that each expert has a fixed number of replicas deployed globally, so as to distribute the traffic of popular experts through a multi-replica mechanism.

[0266] To ensure that all copies from the same expert use the same precision, a precision consistency constraint can be introduced. Let... Experts Whether to adopt For bit precision, we have: ,and In the formula, This means that each expert corresponds to only one level of quantification precision. Indicates if experts In the equipment Deployed on If the bit copy is used, then the expert's overall precision selection must be... This constraint ensures consistency between copy placement and precision selection.

[0267] Furthermore, in step 2, constraints can be applied to the relationship between the total number of replicas and the equipment's carrying capacity. Assume the equipment... The maximum number of deployable experts is Then we have:

[0268]

[0269] By introducing this constraint, it is possible to further prevent a device from experiencing excessive runtime scheduling and caching pressure due to deploying too many copies.

[0270] In one embodiment, the initial construction phase may compute an incremental target value for each candidate device-precision pair:

[0271] In step 3, the pruned greedy search method can be divided into two parts: an initial construction stage and a local refinement stage. In the initial construction stage, replicas sorted by expert popularity are sequentially evaluated in the candidate set. In the local refinement stage, adjacency searches are performed only on the tailing device set and the error-dominant expert set, thereby reducing search complexity while ensuring search quality, as follows:

[0272]

[0273] In the formula, Deployment experts The One copy to the device And adopt precision The resulting incremental total cost This represents the corresponding latency proxy increment. This represents the corresponding quantization error penalty increment. The smaller the increment cost, the more suitable the device-precision pair is as the deployment location for the current replica.

[0274] In one embodiment, the trailing equipment can be assembled. The identification rule is defined as selecting the top devices in terms of latency contribution. The equipment, namely:

[0275]

[0276] In the formula, Indicates equipment Contribution to overall latency This indicates selecting the previous device from all devices. The device that contributes the most to latency. By performing a local search only on the trailing device, unnecessary migration and accuracy adjustment attempts can be reduced.

[0277] Error-driven expert set The identification rule is defined as selecting the top [number] with the largest product of heat and quantization error. One expert, namely:

[0278]

[0279] In the formula, Experts Quantization error under the current precision configuration This indicates that the top experts were selected from all experts. The expert who contributes the most to the error. This rule prioritizes local search for popular experts who are most sensitive to overall accuracy.

[0280] Furthermore, in the local refinement stage, if the candidate operation Corresponding change in objective function If the value is less than zero, the update is accepted. Otherwise, the current configuration remains unchanged. The iterative process can continue until a certain number of consecutive searches no longer produce negative increments, or until the preset maximum number of iterations is reached.

[0281] In step 4, in addition to local priority and remote weighted round-robin polling, it is also possible to determine whether the local replica is overloaded. Let... For experts The local congestion threshold is set so that when the local replica queue length exceeds this threshold, remote replica selection can be triggered, as shown below. , Indicates the current computing device Top experts Local copy at time The queue length. This additional check allows for the activation of remote replica offloading mechanisms even when local replicas exist and are severely congested.

[0282] In one embodiment, the candidate remote replica set can be denoted as For any token to be routed, only... The algorithm calculates dynamic weights for each candidate device and constructs a round-robin scheduler based on these weights, thereby avoiding indiscriminate searching of all devices. It can also perform sorting at fixed time granularities. , and The update is performed, and the updated weights are written to the replica routing table. This replica routing table is maintained by the coordinating node or by each device maintaining a synchronized copy locally, to support low-overhead runtime scheduling.

[0283] Furthermore, in one embodiment, if a remote replica is unable to participate in scheduling due to device failure, link anomaly, or unavailability, the replica is temporarily removed from the candidate set and reinstated in the weighted round-robin once it recovers. This approach improves the runtime routing mechanism's adaptability to fluctuations in real-world heterogeneous environments.

[0284] In one embodiment, the method of the present invention is particularly suitable for hybrid expert model deployment scenarios with large parameter scales, a large number of experts, and significant differences in device capabilities. For systems where popular experts are clearly concentrated, heterogeneous devices have significant performance differences, and cross-device link conditions are inconsistent, the present invention, through a combination of offline joint optimization and online dynamic routing, can more effectively suppress the tailing effect caused by weak devices and reduce communication bottlenecks caused by low-bandwidth links.

[0285] If the model service scenario requires higher accuracy, then the accuracy can be increased. The value of is chosen to make the joint optimization pay more attention to the quantization error penalty term during the solution process. If the model service scenario has higher throughput requirements, it can be appropriately reduced. The value of makes the joint optimization more biased towards reducing latency surrogate terms. Therefore, this invention can achieve flexible and adjustable trade-off control between inference latency, throughput, and model accuracy according to the performance requirements of different application scenarios.

[0286] If the runtime detects the effective bandwidth of a certain link If the decrease is significant, the communication tailing weight coefficient can be temporarily increased without changing the offline copy configuration. Alternatively, the congestion cost of the destination device on the corresponding link in the remote route can be increased, thereby reducing remote scheduling requests across that link. Once the link returns to normal, the original weight settings are restored. This approach improves the adaptability of the invention to scenarios involving link jitter and localized communication degradation.

[0287] If the system observes that a certain device is under high load for an extended period while other devices have low utilization, the system can increase the priority of migrating out copies of popular experts from the high-load device in subsequent offline planning cycles, and correspondingly increase the priority of accepting less popular experts from low-load devices. This approach allows the system to gradually approach a more balanced global deployment state across multiple offline planning cycles.

[0288] In summary, this invention effectively solves the problems of memory bottleneck, unbalanced computational load, and communication tail in MoE models under heterogeneous environments. While significantly reducing inference latency, it ensures model accuracy through expert activation frequency-aware quantization, and significantly improves the deployment efficiency and throughput of heterogeneous clusters.

[0289] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A hybrid expert model inference joint deployment optimization method for heterogeneous GPU environments, characterized in that, The method includes the following steps: Step 1: Construct a hybrid expert model distributed inference system on a heterogeneous GPU cluster, collect the memory budget, computing power and inter-device bandwidth information of each GPU device, and count the activation frequency of each expert at each layer based on the offline inference trajectory. Step 2: Based on the activation frequency of experts, and with the goal of minimizing the weighted sum of the inference latency surrogate term and the quantization error penalty term, construct a joint optimization model of expert copy placement and mixed precision quantization that includes memory constraints. Step 3: Based on the joint optimization model, solve for the replica placement configuration, quantization bit width and replica number of each expert on each GPU device to obtain the offline replica configuration result; Step 4: During online inference, based on the offline replica configuration results, a replica routing mechanism combining local priority and load-aware weighted round-robin is used to complete the token distribution and execution.

2. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 1, characterized in that, Step 1 involves building a hybrid expert model distributed inference system on a heterogeneous GPU cluster, specifically including: The hybrid expert model is deployed in a cluster consisting of multiple heterogeneous GPU devices, denoted as set. elements in Each GPU device has a corresponding video memory budget. ; During the inference process, the shared sublayer processes the input token to obtain the token hidden state, and then the router selects the top-k target experts for each token; The target experts are distributed across multiple GPU devices in a parallel manner, so that the target expert corresponding to the token can be located on both a local GPU device and a remote GPU device. When the target experts are located on different GPU devices, the token hidden state is distributed through cross-device data exchange, and the expert output is sent back to the initiating device after the expert's calculation is completed.

3. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 1, characterized in that, Step 1 involves calculating the activation frequency of each expert at each layer based on the offline inference trajectory, specifically including: Acquire offline inference trajectory data and count the number of times each expert is activated in each layer; Calculate the activation frequency of each expert within their respective layer; The expert activation frequency is then expressed as: In the formula, Indicates the first Layered experts activation frequency, Indicates the first Layered experts Number of times it is activated Indicates the first A group of experts at the level Indicates the first Layer of experts The number of times it has been activated.

4. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 1, characterized in that, The joint optimization model of expert copy placement and mixed-precision quantization in step 2 is as follows: For any expert Let the total number of its copies be . Then the following conditions are met: And for the same expert The constraint for all replicas is to use a unique quantization bit width, so that the expert does not have multiple different quantization bit widths among all replicas; in, Experts One Is the bit copy placed on the GPU device? superior; This represents a collection of GPU devices.

5. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 4, characterized in that, The objective function of the joint optimization model described in step 2 is expressed as follows: : In the formula, This indicates the placement and quantitative configuration of expert copies. To bring together all experts, For the set of candidate quantization bit widths, Indicates the surrogate term for inference delay. This represents the quantization error penalty term. This represents the tradeoff coefficient between latency and quantization error; The memory constraint is, for any GPU device The total memory usage of all expert copies and other model parameters deployed on this device does not exceed its video memory budget, expressed as: In the formula, Experts exist Parameter storage requirements under bit quantization Indicates equipment The model parameters and runtime overhead, excluding expert copies, are as follows. Indicates equipment The video memory budget.

6. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 5, characterized in that, The quantization error penalty item Represented as: In the formula, Experts Achieving full precision Experts of Bit quantization implementation, Indicates routing to experts in offline tracks The set of hidden states Indicates configuration Next Layer experts The quantization bit width used, Experts Quantization error, Indicates the first Layered experts Activation frequency; Represents the square of the L2 norm. Indicates input to expert The hidden state.

7. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 5, characterized in that, The inference delay proxy item This includes computational delay proxy items and communication delay proxy items, represented as: To calculate the delay surrogate term, it is represented as: in, In the formula, Indicates deployment on GPU device Experts Aggregated service capabilities Indicates deployment on GPU device Experts Aggregated service capabilities Experts of Bit copies on GPU devices Service speed on; Experts Assigned to GPU devices The proportion of traffic, Experts Overall activation frequency, Indicates the first Layers on GPU devices The computational load on top, Indicates the first The average computational load of the layer across all GPU devices; This represents the communication latency weighting coefficient, used to balance the impact of unbalanced computational load and cross-device communication tailing on overall latency. Indicates the first A collection of experts at the level; For communication delay proxy items, it is represented as: In the formula, Indicates in configuration The following equipment Issued and in the device The expected rate of token processing is as follows: Indicates GPU device With GPU devices The effective bandwidth between them.

8. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 5, characterized in that, Step 3 uses a pruning greedy search method to solve for the replica placement configuration of each expert on each GPU device, quantizing the bit width and the number of replicas. The specific process includes: Step 3-1: Divide all experts into groups to obtain the expert group set. Experts within the same expert group share the quantization bit width. To reduce the quantization search space; Step 3-2: Sort experts by activation frequency from high to low, and assign copies to experts accordingly. In each assignment, select the proxy that can reduce inference latency. The smallest GPU device is the target deployment device; Step 3-3: After obtaining the initial deployment results, identify the trailing device set and the error-dominant expert set, and perform a local search, focusing only on the top experts in the trailing device set. The top GPU devices and error-dominant expert set Several experts conduct copy migration or bit width reduction searches; based on the current configuration, the computational and communication loads of all devices are analyzed, and a set of devices whose contribution to overall latency is greater than a first preset threshold is selected to form a tailing device set. Based on the contribution of the quantified error penalty term, experts whose impact on overall accuracy is greater than a preset second threshold are selected to form an error-dominant expert set. ; Steps 3-4: When a candidate move causes a change in the objective function... When the candidate move is accepted, the current configuration is updated, and the local search continues until no candidate move is found that causes the objective function to continue to decrease.

9. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 1, characterized in that, The replica routing mechanism combining local priority and load-aware weighted round-robin described in step 4 satisfies the following: when a local replica of the target expert exists, local execution is always given priority; when remote execution is the only option, tokens are preferentially distributed to GPU devices with a service rate greater than the first preset threshold and a prediction queue length less than the second preset threshold, so as to achieve cross-device load balancing while maintaining the local priority principle; wherein the first preset threshold and the second preset threshold are customizable.

10. The method for joint deployment optimization of hybrid expert model inference in heterogeneous GPU environments according to claim 9, characterized in that, The replica routing mechanism, which combines local priority with load-aware weighted round-robin, specifically includes: For each input token, first determine whether there is a local copy of the target expert on the current GPU device. If there is a local copy, prioritize routing to the local copy for execution. If there is no local copy, or the local execution load exceeds the preset conditions, perform load estimation based on the queue length and service rate of the candidate remote copies, and use a load-aware weighted round-robin strategy to select the target remote copy.

Images