A heterogeneous cost-aware expert weight distribution method and device for a super-large-scale hybrid expert model

Abstract

The application discloses a heterogeneous cost-aware expert weight distribution method and system for an ultra-large-scale hybrid expert model. The method comprises the following steps: generating and updating a full-network central processor memory cache state table in real time; constructing a heterogeneous transmission cost matrix according to the full-network central processor memory cache state table; obtaining an integer linear programming model with group limit constraints; performing double-target optimization solving on the integer linear programming model to obtain an optimal expert-computing card distribution scheme and an expert group-physical node binding scheme; and pulling expert weights from corresponding source positions by each computing card according to the optimal expert-computing card distribution scheme and the expert group-physical node binding scheme. The application solves an ultra-fast recovery scheme which not only satisfies load balancing but also strictly obeys group limit routing by using a mixed integer linear programming, so that the EPLB service suspension time of a large-scale cluster reaches the theoretical optimum, and the normal reasoning performance is absolutely not affected.

Description

Technical Field

[0001] This application relates to the field of large model technology, specifically to a heterogeneous cost-aware expert weight allocation method and system for ultra-large-scale hybrid expert models. Background Technology

[0002] Top-tier large language models such as DeepSeek-V3 widely employ large-scale hybrid expert model (MoE) architectures and expert parallelism (EP) strategies. To reduce inter-node communication overhead, advanced large model systems introduce group-limited expert routing, which stipulates that a group of experts should be placed on the same physical node to reduce all-to-all communication traffic.

[0003] Because each computing card has limited computing power, the system needs to periodically perform expert parallel load balancing (EPLB) based on the real-time token load of each expert, migrating the weights of experts with high loads between different computing cards. Simultaneously, when a hardware failure (rank fault) occurs in the multi-card cluster, causing the loss of some expert weights, EPLB must also be triggered immediately for fault tolerance recovery.

[0004] Currently, mainstream distributed inference frameworks (such as vLLM, SGLang, etc.) mainly adopt the following methods when handling EPLB and fault tolerance: Undifferentiated topology assumption: When calculating the cost of weight migration, the actual bandwidth differences of intra-node GPU interconnects (such as NVLink), inter-node network communication (such as RDMA / IBGDA), and processor bus (PCIe) are not distinguished.

[0005] Inefficient use of CPU RAM: Large distributed inference systems typically have redundant CPU memory but insufficient video memory. Existing systems do not fully utilize CPU memory to reduce fault recovery time or normal EPLB speed.

[0006] The disadvantages of existing technology are as follows: 1. Long service downtime: During weight migration and transmission, the downtime of the entire inference service depends on the last compute card to complete weight reception. Existing technologies often cause a single node to become a system bottleneck due to pulling too many weights across the network or from the disk.

[0007] 2. Performance degradation due to violation of group-restricted routing: When a failure occurs, existing algorithms may assign experts from the same group to different nodes in order to recover lost experts as quickly as possible. While this achieves fault tolerance, it violates the group-restricted routing rules, leading to a large amount of cross-node network communication during subsequent normal inference, resulting in performance degradation.

[0008] 3. Low memory (CPU RAM) utilization: Since the EPLB results after a failure are often different from the normal state, traditional passive local memory backup often faces the dilemma of "no required experts in local memory" when a failure occurs, and can only degenerate into slow cross-node network pull or even reading from disk. Summary of the Invention

[0009] The purpose of this invention is to provide a heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models, which at least solves one of the above-mentioned technical problems.

[0010] One aspect of the present invention provides a heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models, used in a distributed inference cluster comprising multiple physical nodes, each physical node being configured with at least one computing card and central processing unit memory, the heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models comprising: Step 1: In the background of normal model distributed inference, perform predictive memory cache scheduling asynchronously and periodically to generate and update the memory cache status table of the entire network's central processing unit in real time. Step 2: Construct a heterogeneous transmission cost matrix based on the network-wide central processing unit memory cache status table; Step 3: Obtain an integer linear programming model with group constraints. This model includes expert-computing card allocation variables, expert group-physical node binding variables, and the longest weight reception time variable across the entire network. It also includes constraints on the number of expert replicas, mandatory group affinity, memory and computational load, and the maximum time-consuming envelope constraint. Specifically, under the mandatory group affinity constraint, if a computing card loads an expert, the entire expert group to which that expert belongs is bound to the physical node where that computing card resides. Under the maximum time-consuming envelope constraint, the total weight reception time of any computing card does not exceed the longest weight reception time variable across the entire network. Step 4: With minimizing the longest weighted reception time across the entire network as the primary objective and minimizing the total global transmission volume as the secondary objective, perform bi-objective optimization on the integer linear programming model to obtain the optimal expert-computing card allocation scheme and expert group-physical node binding scheme; Step 5: Based on the optimal expert-computing card allocation scheme and the expert group-physical node binding scheme, each computing card pulls expert weights from the corresponding source location.

[0011] Optionally, step 1 includes: Step 11: The distributed inference cluster records the historical word processing volume of each expert in each layer in real time in the background through a sliding window, and uses the exponential moving average algorithm to calculate the expected load heat of each expert. When it is predicted that the total expert load of a certain computing card is about to exceed its computing power threshold, the expert with the fastest load growth on the computing card is marked as a high-probability migration candidate expert. Step 12: The distributed inference cluster calculates the node affinity score of each high-probability migration candidate expert to all surviving physical nodes in the cluster; Step 13: The distributed inference cluster selects the K target nodes with the highest node affinity scores for each high-probability migration candidate expert. During normal inference, the idle and fragmented bandwidth of the cross-node network is used to copy the weights of the candidate experts to the central processing unit memory of the K target nodes in advance and asynchronously. Step 14: The distributed inference cluster performs local mirroring and ring-shaped cross-node disaster recovery backup; Step 15: The distributed inference cluster generates and updates the network-wide central processing unit memory cache status table in real time, recording the storage location of all expert weights for prefetching, local mirroring, and ring backup.

[0012] Optionally, the node affinity score in step 12 includes three dimensions: group constraint affinity, load complementarity, and network topology cost. Specifically, if the logical group to which the high-probability migration candidate expert belongs has been assigned to the target physical node, the score for the group constraint affinity dimension increases; if the target physical node's current total computing power load is decreasing and has sufficient computing power margin, the score for the load complementarity dimension increases; and if the underlying physical network bandwidth from the physical node currently holding the high-probability migration candidate expert to the target physical node is sufficient, the score for the network topology cost dimension increases.

[0013] Optionally, step 2 includes: Step 21: The distributed inference cluster scans in real time the status of all surviving computing cards in the current cluster, the central processing unit memory status of all physical nodes, and the global persistent storage status to obtain the current storage location of each expert; Step 22: Based on the expert storage location obtained in Step 21, and combined with the communication topology layer between the expert and the target computing card, set the transmission cost weight according to the principle of increasing cost, and calculate the initial transmission time baseline value for any expert to be transmitted to any surviving computing card. Step 23: Query the network-wide central processing unit memory cache status table. If the weight of the corresponding expert has been pre-fetched in the central processing unit memory of the physical node where the target computing card is located, then the initial transmission time baseline value of the expert to all computing cards in the physical node in Step 22 is reduced from the cross-node network transmission cost to the same-node high-speed serial computer expansion bus transmission cost to obtain the final heterogeneous transmission cost matrix.

[0014] Optionally, the integer linear programming model with set constraints in step 3 includes:

[0015] Define the expert-calculation card allocation variable as follows: , Indicates whether expert e is allocated to the video memory of computing card r; defines the expert group-physical node binding variable as... , Indicates whether expert group g is entirely bound to physical node n; defines the longest weighted reception time variable in the entire network as... , representing the maximum weight reception time among all computing cards; where e∈Expert set, r∈Computing card set, g∈Expert group set, n∈Physical node set; The expert replica number constraint is expressed by the formula. Obtain, among which The number of copies that expert group g needs to make; The forced group restriction affinity constraint is expressed by the formula Obtain, among which If r ∈ g and r ∈ n, ensure that if the computation card r loads expert e, then expert group g must be bound to node n; The constraints on video memory and computing load are expressed by the formula. And the computational load limit calculation based on the number of historical word units, where This is the maximum capacity of a single graphics card's video memory. The maximum time-consuming envelope constraint is expressed by the formula... Calculation and acquisition, ensuring that the total weight reception time for any computation card r does not exceed .

[0016] Optionally, the objective function for solving the bi-objective optimization problem is: α is the weight coefficient corresponding to minimizing the longest weight reception time in the entire network, and β is the weight coefficient corresponding to minimizing the total global transmission volume. α > β, so as to prioritize suppressing the longest weight reception time in the entire network, and then penalize the total global transmission volume.

[0017] Optionally, setting the transmission cost weight according to the principle of increasing cost includes: If the expert is already in the target computing card's video memory, the transmission cost weight is 0; If the expert uses the memory of other surviving computing cards in the same node, the first transmission cost weight corresponding to the NVLink interconnect is used; If the experts are in the same node's central processing unit memory, they will use the second transmission cost weight corresponding to the PCIe bus; If the expert uses the third transmission cost weight corresponding to the cross-node RDMA / IBGDA network in the video memory or central processing unit memory of other nodes; If the expert is lost globally, it needs to be loaded from the global persistent storage, using the corresponding fourth transmission cost weight for disk reading; the first transmission cost weight < the second transmission cost weight < the third transmission cost weight < the fourth transmission cost weight.

[0018] Optionally, the distributed inference cluster performs local mirroring and ring-shaped cross-node disaster recovery backup, including: Local Mirror Backup: The expert weights held by each computing card in the current physical node are mirrored and backed up to the central processing unit memory of this node. In the event of a single card failure, other computing cards in the same node can pull the corresponding expert weights from the local central processing unit memory via the PCIe bus to take over. Ring-shaped cross-node disaster recovery backup: The physical nodes in the cluster are divided into a logical ring topology. By utilizing the idle network bandwidth during normal inference, the CPU memory of the Nth physical node asynchronously backs up the expert group weights of the N-1th physical node. In the event of a whole node failure, the Nth physical node can directly complete the expert takeover based on the backup in its local CPU memory without having to reload data from global persistent storage.

[0019] Optionally, the heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models further includes:

[0020] When the expert weight redistribution is triggered and the solution is completed, the CPU memory cache that has been hit is read into the corresponding computing card's video memory; the redundant prefetch cache that has not been hit is silently cleaned up by a background process with the lowest node affinity score as the priority elimination mechanism, or by the least recently used (LRU) mechanism, to release CPU memory space.

[0021] This application also provides a heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models. The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models includes a distributed inference cluster with multiple physical nodes. Each physical node is configured with at least one computing card and a central processing unit memory. The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models is used to obtain the heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described above.

[0022] This application proposes a weight allocation algorithm that finely models the heterogeneous transmission time. It uses mixed-integer linear programming to solve a fast recovery scheme that not only satisfies load balancing but also strictly adheres to group-restricted routing. Furthermore, it employs a multi-level CPU-memory pre-warming caching strategy to transform passive backup into active pre-fetching, enabling the EPLB service pause time of large-scale clusters to reach the theoretical optimal level without dragging down normal inference performance. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a heterogeneous knowledge-based expert weight allocation method for ultra-large-scale hybrid expert models according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram illustrating the cost assignment principle of an embodiment of this application, which is based on the increasing topological hierarchy.

[0025] Figure 3 This is a schematic diagram of the periodic asynchronous execution of memory scheduling in one embodiment of this application.

[0026] Figure 4 This is a schematic diagram of an additional disaster recovery backup in one embodiment of this application.

[0027] Figure 5 This is a timing diagram illustrating the system performing ultra-fast weight migration in one embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models, as shown, is used in a distributed inference cluster containing multiple physical nodes, each physical node being configured with at least one computing card and central processing unit memory. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models includes: Step 1: In the background of normal model distributed inference, perform predictive memory cache scheduling asynchronously and periodically to generate and update the memory cache status table of the entire network's central processing unit in real time. Step 2: Construct a heterogeneous transmission cost matrix based on the network-wide central processing unit memory cache status table; Step 3: Obtain an integer linear programming model with group constraints. This model includes expert-computing card allocation variables, expert group-physical node binding variables, and the longest weight reception time variable across the entire network. It also includes constraints on the number of expert replicas, mandatory group affinity, memory and computational load, and the maximum time-consuming envelope constraint. Specifically, under the mandatory group affinity constraint, if a computing card loads an expert, the entire expert group to which that expert belongs is bound to the physical node where that computing card resides. Under the maximum time-consuming envelope constraint, the total weight reception time of any computing card does not exceed the longest weight reception time variable across the entire network. Step 4: With minimizing the longest weighted reception time across the entire network as the primary objective and minimizing the total global transmission volume as the secondary objective, perform bi-objective optimization on the integer linear programming model to obtain the optimal expert-computing card allocation scheme and expert group-physical node binding scheme; Step 5: Based on the optimal expert-computing card allocation scheme and the expert group-physical node binding scheme, each computing card pulls expert weights from the corresponding source location.

[0030] In this embodiment, step 1 includes: Step 11: The distributed inference cluster records the historical word processing volume of each expert in each layer in real time in the background through a sliding window, and uses the exponential moving average algorithm to calculate the expected load heat of each expert. When it is predicted that the total expert load of a certain computing card is about to exceed its computing power threshold, the expert with the fastest load growth on the computing card is marked as a high-probability migration candidate expert. Step 12: The distributed inference cluster calculates the node affinity score of each high-probability migration candidate expert to all surviving physical nodes in the cluster; Step 13: The distributed inference cluster selects the K target nodes with the highest node affinity scores for each high-probability migration candidate expert. During normal inference, the idle and fragmented bandwidth of the cross-node network is used to copy the weights of the candidate experts to the central processing unit memory of the K target nodes in advance and asynchronously. Step 14: The distributed inference cluster performs local mirroring and ring-shaped cross-node disaster recovery backup; Step 15: The distributed inference cluster generates and updates the network-wide central processing unit memory cache status table in real time, recording the storage location of all expert weights for prefetching, local mirroring, and ring backup.

[0031] Step 1: Predictive memory cache scheduling: See Figure 3 This step is a predictive memory cache scheduling process executed asynchronously and periodically in the background. It runs entirely in the background of normal model distributed inference, utilizing idle CPU cycles for scheduling, without interfering with normal inference tasks in the foreground. Its core purpose is to fully utilize idle CPU memory resources and free network bandwidth in the distributed inference cluster, providing a large number of low-latency local cache hits for the subsequent construction of the heterogeneous transmission cost matrix. This transforms traditional passive fault backup and load migration preparation into proactive predictive prefetch caching, thereby significantly reducing the time required for subsequent expert weight reallocation and fault recovery. The specific execution process of this step is as follows: Step 11: Multi-dimensional load trend monitoring and candidate expert tagging The system maintains a sliding time window in the background, which records the historical word processing volume of each expert e in each layer of the model network in real time, thereby capturing the dynamic changes in expert load under the MoE model.

[0032] To smooth out random fluctuations in historical data and more accurately predict future load trends, the system uses the Exponential Moving Average (EMA) algorithm to weight the historical processing volume data of each expert and calculate the expected load intensity of each expert in the future, thereby characterizing the load growth trend of that expert.

[0033] When the system identifies, based on the above predictions, that the total expert load on a computing card is about to exceed the card's computing power threshold, it indicates that the card is at risk of overload and some experts need to be migrated out in the next Expert Parallel Load Balancing (EPLB) cycle to achieve load balancing. At this point, the system will mark the expert with the fastest load growth on that computing card as a high-probability migration candidate. This expert is the most likely to be migrated out in the subsequent load balancing process, and the system will perform cache prefetching preparation for this expert in advance.

[0034] Step 12: Quantitative calculation of node affinity score S(e,n)

[0035] For each high-probability migration candidate expert e identified, the system calculates a node affinity score S(e,n) for all surviving physical nodes n in the cluster. This score assesses the rationality and suitability of migrating the candidate expert to the corresponding node, incorporating a comprehensive consideration of three dimensions:

[0036] Group Affinity Dimension: Since this scheme needs to maintain the optimization effect of group-limited routing, experts in the same group should be clustered within the same physical node as much as possible to reduce cross-node communication overhead during subsequent normal inference. Therefore, if the logical group g to which the high-probability migration candidate expert e belongs has already been assigned to the target node n, it means that migrating the expert to that node can maintain the group-limited clustering characteristic and will not disrupt the original communication optimization topology. Therefore, the score for this dimension will increase accordingly.

[0037] Load Complementarity Dimension: In order to ensure the load balance of the cluster after migration, the system will evaluate the load status of the target node n. If the total computing power load of node n is currently decreasing and there is sufficient computing power margin, it means that the node can bear the load of the candidate expert and there will be no overload of the target node after migration. Therefore, the score of this dimension will increase accordingly.

[0038] Topology Cost dimension: Prefetching operations require the use of idle network bandwidth to transmit expert weights. In order to avoid prefetching transmission interfering with normal inference, the system will evaluate the underlying physical network bandwidth status from the node currently holding expert e to the target node n. If the network bandwidth between the two points is sufficient, it means that prefetching transmission can be completed without crowding out normal inference bandwidth. Therefore, the score of this dimension will increase accordingly.

[0039] In this embodiment, the node affinity scoring method is as follows: For each expert e marked as a high-probability migration candidate, the system calculates a node affinity score S(e, n) for all surviving physical nodes n in the cluster. This score is used to evaluate the priority of pre-fetching the expert's weight into the target node's CPU memory. The calculation formula is as follows:

[0040] Wherein, α, β, and γ are preset weighting coefficients (satisfying α + β + γ = 1), which can be adjusted according to the actual emphasis of the cluster on communication or computing power. The calculation method for each sub-score is as follows: (1) Group Restricted Affinity Score : This item measures whether the requirement for clustering of experts in the same group can be met after expert e migrates to node n: ; Variable meaning: g represents the expert group to which expert e belongs; This represents the number of experts belonging to expert group g in the current node n. This represents the total number of experts in expert group g.

[0041] Objective: The more experts in the same group within the target node, the higher the score. This ensures that even if a migration occurs, the performance of group-limited routing is not compromised.

[0042] (2) Load Idle Score : This parameter is used to assess whether the target node n has sufficient computing power margin to accommodate new experts. ; ; Variable meaning: This represents the real-time computing load currently borne by node n (in terms of the number of tokens processed). τ is the rate of change of load at node n over the past sliding time window; τ is the prediction time step constant. The predicted future load of the target node; This represents the maximum word processing capacity threshold when the node is fully loaded.

[0043] Target: The value range is [0, 1]. If node n currently has a low load and it is trending downwards ( ), It will become smaller, making As the value approaches 1, the score increases. Therefore, the more idle a node is, the higher its score, which helps prevent experts from migrating to congested nodes.

[0044] (3) Network topology bandwidth score

[0045] This item is used to evaluate the bandwidth cost of transmitting weights from the expert's current node m to the target node n: ; Variable meaning: This represents the actual available idle bandwidth between nodes m and n, as obtained through the network monitoring component. This represents the maximum physical bandwidth between nodes m and n.

[0046] Objective: The higher the percentage of idle bandwidth, the higher the score. This ensures that the system only utilizes idle network bandwidth for asynchronous prefetch transmission. Step 13: Multi-Optimal Prefetch Caching In distributed inference clusters, the memory capacity of a single computing card is usually quite valuable (e.g., typically 80GB), while the CPU memory capacity of physical nodes is usually more abundant and cheaper (reaching 1TB-2TB). Therefore, the system can take advantage of the large capacity of CPU memory to perform redundant prefetch caching, avoiding the limitation of inaccurate single-target prediction.

[0047] Based on the calculated node affinity score S(e,n), the system selects the Top-K target nodes with the highest scores for candidate expert e (K can be 3 in this embodiment). Then, utilizing the idle and fragmented bandwidth of the cross-node network (RDMA / IBGDA) during normal inference, the weight of expert e is copied to the CPU memory of these K target nodes in advance and asynchronously.

[0048] In this way, the system prepares K possible optimal solution caches for possible EPLB load balancing scenarios in the future. Even if a sudden node failure (rank fault) occurs later, this redundant multi-optimal solution cache can greatly improve the cache hit rate during fault tolerance and avoid the need to reload data from slow persistent storage.

[0049] Step 14: Two-level disaster recovery backup See Figure 4 In addition to predictive prefetching for high-probability migration experts, the system also performs two levels of disaster recovery backups to prepare for unexpected hardware failures: Local Mirroring Cache: For the current physical node, the system mirrors and backs up the expert weights currently held by all compute cards within that node to the CPU memory of that node. With this backup, if a single card fails subsequently, and other surviving compute cards within the same node need to take over the expert load of the failed card, there is no need for cross-node network transmission. The takeover can be completed directly by pulling the corresponding expert weights from the local CPU memory via the high-speed PCIe bus, significantly shortening the recovery time for single-card failures.

[0050] Ring-shaped cross-node disaster recovery caching: The system divides all physical nodes in the cluster into a logical ring topology. Utilizing idle network bandwidth during normal inference, it performs asynchronous cross-node backups: the CPU memory of the Nth physical node silently and asynchronously backs up the expert group weights of the (N-1)th physical node. Through this backup mechanism, if a subsequent failure occurs where the entire physical node goes down, the Nth physical node responsible for backing up that node does not need to reload data from slow global persistent storage (disk). It can directly take over the experts on the failed node based on the existing backup in its local CPU memory, achieving extremely fast recovery from the entire node failure.

[0051] Step 15: Update the CPU memory cache status table across the entire network. After completing the predictive prefetching, local mirror backup, and ring cross-node backup described above, the system will generate and update the network-wide central processing unit memory cache status table in real time. This table will record the storage location of the expert weights for all prefetching, local mirroring, and ring backups, providing a basis for the subsequent construction of the heterogeneous transmission cost matrix, and enabling the system to perceive the cache hit status of each expert in real time.

[0052] The prefetching cache operation in this step directly and inversely affects the decision-making process of the subsequent GPU-side integer linear programming (ILP) solver, forming a two-way collaborative closed loop: When EPLB load balancing or fault tolerance is actually triggered, and the solver begins constructing the heterogeneous transmission cost matrix Cost(e,r), it will first query the aforementioned network-wide CPU RAM cache status table. If the CPU memory of candidate node n has already prefetched the weight of expert e, then allocating the transmission cost Cost(e,r) of expert e to any computing card r within that node will instantly reduce the cost from the originally expensive cross-node network (e.g., W3=10) to the same-node PCIe bus read cost (e.g., W2=3).

[0053] Since the objective function of the planning solver is to minimize the total time and maximum delay (T_{max}), the sharp drop in cost will guide the solver to prioritize assigning expert e to computing cards that have already prefetched that weight. This ultimately forms a closed loop where the CPU pre-paves the way and the GPU algorithm makes its moves accordingly, maximizing the utilization of the prefetch cache and avoiding the waste of prefetch resources.

[0054] Cache lifecycle management (eviction policy): To avoid redundant prefetching caching consuming excessive CPU and memory resources, the system also performs cache lifecycle management in the background: Once the expert weight redistribution is triggered and the solution is completed, the hit CPU memory cache will be read into the GPU memory of the corresponding computing card and used for subsequent inference. The redundant prefetch cache that is not hit will be silently cleaned up by the system through a background process, prioritizing the elimination of the node with the lowest node affinity score or the least recently used (LRU) mechanism, to release the corresponding CPU memory space and make room for the next round of predictive cache scheduling. This cleanup process is executed asynchronously in the background and will not interfere with the normal inference tasks in the foreground.

[0055] In this embodiment, step 2 includes: Step 21: The distributed inference cluster scans in real time the status of all surviving computing cards in the current cluster, the central processing unit memory status of all physical nodes, and the global persistent storage status to obtain the current storage location of each expert; Step 22: According to the expert storage position obtained in step 21, combined with the communication topology level between the expert and the target computing card, the transmission cost weight is set according to the principle of increasing cost, and the initial transmission time reference value for any expert to transmit to any surviving computing card is calculated; Step 23: Query the global central processing unit memory cache state table, and if the weight of the corresponding expert has been prefetched in the central processing unit memory of the physical node where the target computing card is located, the initial transmission time reference value for the expert in step 22 to transmit to all computing cards in the physical node is adjusted down from the cross-node network transmission cost to the same-node high-speed serial computer expansion bus transmission cost, so as to obtain the final heterogeneous transmission cost matrix.

[0056] Step 2: Construction of heterogeneous transmission cost matrix

[0057] This step is executed when expert parallel load balancing (EPLB) or hardware fault tolerance is triggered. The core purpose is to combine the global CPU memory cache state table generated in step 1, quantify the time consumption difference of expert weight transmission under different storage positions and different communication topologies, construct an accurate heterogeneous transmission cost matrix, solve the problem of inaccurate transmission cost modeling caused by the undifferentiated topology assumption in the prior art, and provide an accurate cost basis for subsequent integer linear programming solutions. The specific execution process of this step is as follows: S21: Cluster full-state scanning and expert storage position acquisition The system first scans the full operation state of the current cluster in real time (scans the surviving graphics cards (VRAM), memory (CPU RAM) and storage state of the current cluster in real time), and the scanning content includes: the operation state of all surviving computing cards, the CPU memory state of all physical nodes, and the state of global persistent storage, so as to obtain the current actual storage position of each expert e, and provide a basic basis for subsequent transmission cost calculation.

[0058] S22: Calculation of initial transmission time reference value Cost(e,r) See Figure 2 , based on the expert storage position obtained in step S21, the system combines the communication topology level between expert e and target computing card r, and calculates the initial transmission time reference value Cost(e,r) for transmission of any expert e in the system to any surviving computing card r according to the principle of increasing cost.

[0059] The cost weight assignment rules for different communication topology levels are as follows, all weights satisfy the increasing relationship of 0<W1<W2<W3<W4, so as to match the time consumption difference of different transmission modes: If the expert e is already in the video memory of the target computing card r, it means that no data transmission is required, so the transmission cost weight is 0; If expert e is in the video memory of other surviving computing cards in the same node, it means that the transmission can be completed through the high-speed NVLink interconnect within the node. Therefore, the cost weight is set to W_1 (in this embodiment, the value can be 1). If expert e is in the CPU memory of the same node, it means that the transmission can be completed through the PCIe bus within the node. Therefore, the cost weight is set to W2 (in this embodiment, the value can be 3). If expert e is located in the video memory or CPU memory of other nodes, it means that the transmission needs to be completed through the cross-node RDMA / IBGDA network. Therefore, the cost weight is set to W3 (in this embodiment, the value can be 10). If expert e is lost globally (e.g., the physical node to which it belongs is completely down), it means that the data needs to be reloaded from the global shared storage / disk. Therefore, the cost weight is set to W_4 (in this embodiment, the value can be 100).

[0060] S23: Cost Adjustment Based on Cache State and Final Matrix Generation After calculating the initial transmission cost, the system queries the network-wide CPU memory cache status table generated in step 1 to adjust the transmission cost of the corresponding expert, thereby achieving a two-way linkage between prediction caching and planning solution. If it is found that the weight of expert e has been prefetched in the CPU memory of the physical node n where the target computing card r is located, the system will transfer the expert e to the initial transmission time baseline value Cost(e,r) of all computing cards in that node, reducing it from the original cross-node network transmission cost W3 to the same-node PCIe bus transmission cost W2.

[0061] Through the above adjustments, a complete heterogeneous transmission cost matrix is ​​finally obtained. This matrix accurately reflects the differences in transmission time under different topologies and different cache hit conditions, providing accurate cost input for subsequent integer linear programming solutions.

[0062] In this embodiment, step 3: Obtain an integer linear programming model with group constraints. The integer linear programming model with group constraints includes expert-computing card allocation variables, expert group-physical node binding variables, and the longest weight reception time variable across the entire network, as well as expert replica number constraints, mandatory group constraint affinity constraints, memory and computing load constraints, and maximum time-consuming envelope constraints. Among the mandatory group constraint affinity constraints, if a computing card loads an expert, then the entire expert group to which the expert belongs is bound to the physical node where the computing card is located. Among the maximum time-consuming envelope constraints, the total weight reception time of any computing card does not exceed the longest weight reception time variable across the entire network. This step is performed after the heterogeneous transmission cost matrix is ​​constructed. Its core purpose is to transform the expert weight redistribution problem into an integer linear programming problem. By introducing specific constraints such as group restrictions, it ensures that the allocation result satisfies both the hardware limitations of memory and computing power, maintains the optimization characteristics of group-restricted routing, and eliminates the bottleneck of the "weakest link" effect. This addresses the problem in existing technologies where fault recovery disrupts group routing, leading to performance degradation in inference. The specific details of this step are as follows: Definition of decision variables The system first defines the decision variables for the integer linear programming model to quantify the decision states of the allocation: Expert-Calculation Card Assignment Variables : A Boolean variable with a value of {0, 1}, where e belongs to the expert set and r belongs to the computing card set; this variable indicates whether expert e is allocated to the video memory of computing card r. =1 represents allocation. =0 means no allocation.

[0063] Expert Group - Physical Node Binding Variables : A boolean variable with a value of {0, 1}, where g belongs to the expert group set and n belongs to the physical node set; this variable indicates whether the expert group g is bound to the physical node n as a whole. =1 represents binding. =0 means no binding.

[0064] The longest weighted reception time variable in the entire network : A continuous type variable used to represent the maximum time taken to receive expert weights among all computing cards in the entire network, which is the time taken for the shortest plank in the barrel effect, providing a quantitative basis for subsequent bottleneck optimization.

[0065] Preset constant definition The system also defines preset constants for the model as input constraints: Replication requirement constant This indicates the number of replicas that expert group g needs to replicate in order to meet the computing power load requirements of the cluster, ensuring that each expert group has enough instances to handle the cluster's load requests.

[0066] Single card memory limit : Indicates the maximum video memory capacity of a single computing card, used to limit the number of experts that can be loaded on a single card and prevent video memory overflow.

[0067] Model constraint definition Based on the above variables and constants, the system injects the following four types of mathematical constraints into the model to regulate the feasible range of allocation and ensure that the allocation results meet the design requirements: The expert replica count constraint ensures that each expert group has enough replicas to handle the cluster's load requirements. The corresponding constraint formula is: That is, for each expert group g, the total number of physical nodes it is bound to is equal to the number of replicas required by that expert group. Ensure that the number of copies of the expert group fully meets the load requirements.

[0068] The mandatory group-restricted affinity constraint is one of the core constraints of this scheme. It ensures that the allocation result does not violate the optimization characteristics of group-restricted routing, thus avoiding a large amount of cross-node communication during subsequent normal inference. The corresponding constraint formula is: The constraint applies to the following range: e∈g (expert e belongs to expert group g), and r∈n (computation card r belongs to physical node n). The core principle of this inequality is: it mandates that if computation card r wants to load expert e (i.e., ... =1), then the entire expert group g must be bound to the node n where the card is located (i.e. =1). Through this constraint, the solver will automatically assign experts from the same group to the same physical node, which fully maintains the aggregation characteristics of group-restricted routing. It will not destroy the original communication optimization topology due to load balancing or fault recovery, and completely avoids the performance degradation of subsequent inference.

[0069] The memory and compute load constraint is used to ensure that the allocation result does not exceed the hardware capability limit of the computing card, and includes two sub-constraints: Video memory capacity limit: The corresponding constraint formula is: That is, for each computing card r, the total number of experts loaded cannot exceed the maximum memory capacity of a single card. This avoids the problem of video memory overflow.

[0070] Computing load limit: Based on the historical word processing volume of experts, the total computing load of each computing card r is limited to ensure that the load of each computing card after allocation is within its computing power range, and to avoid node overload.

[0071] The maximum time-consuming envelope constraint is used to eliminate the "weakest link" effect, preventing excessively long weight reception times from any single computing card that could slow down the entire redistribution process. The corresponding constraint formula is: That is, for any computing card r in the system, the total transmission time of all the expert weights it needs to receive cannot exceed the longest weight reception time variable in the entire network. This limits the reception time of all computing cards to a certain limit. This provides a foundation for minimizing bottleneck time in the future.

[0072] In this embodiment, step 4: with minimizing the longest weighted reception time of the entire network as the primary objective and minimizing the total global transmission volume as the secondary objective, the integer linear programming model is optimized to obtain the optimal expert-computing card allocation scheme and expert group-physical node binding scheme. This step is performed after the integer linear programming model is built. Its core objective is to perform a bi-objective optimization solution on the model built in step 3: first, minimizing the bottleneck time of reallocation; second, minimizing the total global transmission volume to avoid the problem of large global weight shifts. Ultimately, it yields the optimal expert-computing card allocation scheme and expert group-physical node binding scheme, achieving extremely fast reallocation and fault recovery. The specific details of this step are as follows: Objective function of bi-objective optimization The system sets a dual-objective optimization function to achieve a hierarchical optimization logic that prioritizes minimizing bottleneck time and then minimizes total transmission volume. The objective function is: ; Where α is the weight coefficient corresponding to minimizing the longest weighted reception time in the entire network, and β is the weight coefficient corresponding to minimizing the total global transmission volume, and the relationship α > β is satisfied to ensure that the primary objective has a higher priority than the secondary objective:

[0073] A larger α weight will cause the solver to prioritize suppressing the longest weight reception time in the entire network, which means prioritizing the compression of the weakest link in the chain, achieving extremely fast redistribution and fault recovery. A smaller beta weight allows the solver to minimize the total global data transfer while satisfying the primary objective, penalizing unnecessary weight transfers and avoiding resource waste.

[0074] With the help of the heterogeneous transmission cost matrix, the objective function guides the solver to automatically select the optimal allocation strategy: in the face of single-point failure or uneven local load, the solver will prioritize allocating the experts that need to be migrated to the surviving computing cards within the same node (via NVLink high-speed transmission) or to the local CPU memory nodes that have already been prefetched and cached (via PCIe transmission), completely eliminating the phenomenon of large-scale global weight migration, avoiding unnecessary cross-node weight migration, and not causing any interference to the operation of other normal nodes.

[0075] For ultra-large-scale computing clusters operating at tens of thousands of kilowatts, when the time required to solve mixed integer linear programming (MILP) problems precisely is too long and cannot meet the requirements for rapid recovery, the system can replace the solution process with a two-layer heuristic greedy algorithm, significantly reducing the solution time while ensuring all core constraints are met. First-level node allocation: Using graph theory matching algorithms, the positions of undamaged expert groups are locked first. The existing allocation of normally operating expert groups is not changed. Only the expert groups on damaged nodes are scheduled to target nodes with corresponding memory backups (RingCaches), thereby minimizing the scope of allocation changes.

[0076] Second-level node allocation: Within the selected target node, a greedy memory allocation strategy is executed to complete the final allocation from expert to computing card, ensuring that the memory capacity and computing load within the node are limited.

[0077] In this embodiment, step 5: according to the optimal expert-computing card allocation scheme and the expert group-physical node binding scheme, each computing card pulls expert weights from the corresponding source location.

[0078] This step is the final execution stage of the entire redistribution process. Its core purpose is to perform asynchronous weight loading and global route updates based on the optimal allocation scheme obtained in step 4, minimizing service downtime while ensuring correct routing for subsequent inference requests, ultimately completing the entire load balancing or fault recovery redistribution process. The specific details of this step are as follows: The system executes differentiated asynchronous loading processes based on the cache hit status of the weights, maximizing the advantages of prefetch caching, minimizing loading time, and reducing the impact of reallocation on normal services. Weight loading that hits the CPU memory cache: For weights that have already been prefetched into the ring cache, that is, a copy of the expert's weights already exists in the CPU memory of the target node, the system loads them directly from the local CPU memory to the target GPU memory. The entire transmission process is completed only through the PCIe bus, without the need for cross-node network transmission, which greatly shortens the loading time and avoids the congestion problem of cross-node network.

[0079] Loading weights that miss the cache: For weights that miss the cache, the system performs asynchronous transmission across nodes, using the cluster's idle network bandwidth to complete the cross-node migration of weights. This transmission process is executed asynchronously, which will not block the execution of normal inference tasks, ensuring that some unaffected inference requests can continue to be processed, and minimizing the impact of reallocation on service availability.

[0080] Once all weights are loaded, the system updates the global expert routing table and synchronously notifies all scheduling nodes in the cluster of the new expert-computing card location information. This ensures that subsequent inference requests can be correctly routed to the new location of the expert based on the updated routing table, avoiding routing errors or request forwarding failures. It also ensures that after the redistribution is completed, all inference requests can successfully hit the corresponding expert.

[0081] After all weights are loaded and routes are updated, all target computing cards complete the weight initialization and report their readiness status to the cluster monitoring system. The cluster monitoring system then lifts the previous service suspension and resumes normal inference requests for the large model. The entire load balancing or fault recovery redistribution process is officially completed.

[0082] In other embodiments, this application also has the following implementation methods: 1. Topology / media alternatives for multi-level memory caching In the predictive memory caching subsystem, in addition to the default logical ring cross-node disaster recovery backup topology, the following two alternatives can be used: Star topology backup: Replace the ring topology of cross-node disaster recovery caching with a star topology, with the central node managing the cache backup of all nodes in a unified manner, simplifying the backup scheduling logic.

[0083] Erasure coding distributed memory striping storage: This solution uses an erasure coding-based distributed storage scheme to distribute the weight slices of a single expert across the CPU memory of multiple nodes. This reduces the CPU computational overhead during recovery in exchange for a lower overall memory storage space usage, thereby improving the cluster's memory utilization.

[0084] In the construction of the heterogeneous transmission cost matrix, in addition to the default static configuration of the weight values ​​(W1, W2, W3, W4) of the heterogeneous topology, a dynamic weight calculation scheme can also be introduced: By introducing a lightweight graph neural network (GNN) or reinforcement learning model, the predicted value of Cost(e, r) can be dynamically output based on the real-time network congestion level and node load status of the current cluster. This allows the cost matrix to adapt to the dynamic changes of the cluster in real time, further improving the optimality of the allocation scheme.

[0085] See Figure 5 The figure above illustrates the timing of rapid weight migration in the event of a single computing card failure or conventional load balancing. This figure clearly demonstrates how the planning and solving subsystem and the predictive memory cache (CPU RAM) work together efficiently to enable the recovery of large model inference services within extremely short latency.

[0086] Step 1: Trigger fault-tolerant interrupt or load balancing Action: When the cluster monitoring system detects that a computing card has crashed or the system load is unbalanced, the system sends a fault-tolerant interrupt or load balancing signal to the planning and solving subsystem.

[0087] This marks the transition of the system from steady-state inference to the expert weight migration phase, at which point the inference service is temporarily suspended.

[0088] Step 2: Query cache status Action: Before deciding how to reallocate expert e, the planning and solving subsystem first sends a query request to the cached state table (CPU RAM) of this node to confirm whether the weight of the expert has been pre-fetched in the background.

[0089] Step 3: Return the cache hit status Action: Thanks to the predictive caching subsystem strategy of this invention, the cache status table returns a hit result: the weight backup of expert e is ready in the CPU memory of the local physical node.

[0090] This is a crucial step in turning a passive situation into a proactive one. Because the system can already prefetch data using idle bandwidth before a failure occurs or load balancing is triggered.

[0091] Step 4: Issue the optimal reallocation instruction Action: The planning and solving subsystem obtains the optimal solution, decides to pass the task baton of expert e to the surviving computing card in the same node, and issues a redistribution instruction to that card.

[0092] Step 5: Read the weights via high-speed bus Action: The survival computing card directly initiates a request to retrieve the expert e weights from the local central processing unit memory (CPU RAM) via the high-speed PCIe bus (or similar high-speed interconnect bus) inside the server.

[0093] Step 6: Transmission complete Action: The CPU RAM transfers the weight data to the VRAM of the survival computing card at high speed via a high-speed bus.

[0094] By avoiding congested, high-latency cross-node networks (such as RDMA / Ethernet), transmission time is significantly reduced.

[0095] Step 7: Fault recovery complete, service restored. Action: The survival computing card completes weight loading and initialization, and reports readiness to the cluster monitoring system. The monitoring system then lifts the service suspension and resumes normal inference requests for the large model.

[0096] This application has the following advantages: Significantly reduces fault-tolerant pause time: Open-source EPLB is based on pure load recalculation, which is prone to cross-node weight storms during fault-tolerant recovery; this invention reduces fault-tolerant pause time by using a cost matrix and By suppressing traffic, the recovery flow is limited to high-speed NVLink and PCIe connections on the same node, significantly reducing service downtime.

[0097] Zero performance loss in normal inference path: Although the primary goal of this invention is fast fault tolerance, by setting group-limited routing as a mathematical constraint, it is ensured that the distribution of experts after recovery still strives to meet the requirements of network communication optimization without causing performance degradation.

[0098] Efficiently utilize heterogeneous storage resources: Fully leverage the previously idle CPU memory space and PCIe bus bandwidth in the distributed cluster. By prefetching prediction, weight transmission time is masked, greatly improving the overall throughput of the inference cluster.

[0099] Completely eliminates the relocation of global weights: Under the influence of heterogeneous costs and dual objective functions, when the solver encounters a single point of failure, it will automatically choose the strategy of "not damaging other normal nodes" and prioritize the allocation of experts on the failed card to surviving resources within the same node. This completely avoids the domino-like migration of global weights and will not cause any interference to normally operating nodes.

[0100] Active prefetching significantly improves fault tolerance hit rate: By prefetching a cache with multiple optimal solutions, multiple possible optimal solutions are prepared in advance for future EPLB states. Even if a sudden node failure occurs, the hit rate of the fault tolerance cache can be greatly improved. This solves the dilemma of "no required experts in local memory" in the event of a failure in traditional passive backup, and avoids degrading to slow cross-node network fetching or even disk reading.

[0101] The bidirectional linkage scheduling loop realizes the bidirectional linkage between CPU memory prefetching and GPU planning and solving. The CPU prefetches weights in advance to reduce transmission costs, and the solver will prioritize the nodes that have been prefetched and cached, forming a closed loop in which the CPU paves the way in advance and the GPU algorithm makes moves accordingly, maximizing the utilization of the prefetch cache.

[0102] Two-tier disaster recovery backup enables rapid takeover: Single-card failure scenario: Local CPU memory expert mirroring allows surviving cards on the same node to directly pull weights via high-speed PCIe without cross-node transmission.

[0103] In the scenario of complete system failure: Ring-shaped cross-node disaster recovery backup allows adjacent nodes to take over quickly by directly utilizing the backup of local CPU memory, without having to reload the weights from slow shared storage / disk.

[0104] Capability to solve ultra-large-scale clusters: For ultra-large-scale clusters with tens of thousands of kilowatts, a two-layer heuristic greedy alternative solution is provided, which solves the problem of excessive time consumption for accurate MILP solution and ensures that redistribution and recovery can be achieved even in large-scale clusters.

[0105] Efficient cache lifecycle management: Automatically cleans up redundant prefetch cache that has missed, and releases memory space by prioritizing the elimination of nodes with the lowest affinity score or by using the LRU mechanism, ensuring efficient recycling of CPU memory resources and preventing waste of memory resources.

[0106] Asynchronous scheduling does not affect normal inference: all prefetching and backup operations are performed asynchronously in the background of normal inference, utilizing CPU idle periods and network idle bandwidth, and will not have any impact on normal inference tasks; the asynchronous transmission of weight loading will also not block inference tasks, minimizing the impact of reallocation on service availability.

[0107] Eliminating the bottleneck of the barrel effect: By constraining the maximum time-consuming envelope, the longest transmission time of the entire network is suppressed, avoiding the transmission bottleneck of a single node from slowing down the recovery speed of the entire cluster, and allowing the EPLB service pause time of large-scale clusters to reach the theoretical optimal level.

[0108] This application also provides a heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models. The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models includes a distributed inference cluster with multiple physical nodes. Each physical node is configured with at least one computing card and a central processing unit memory. The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models is used to obtain the heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described above.

[0109] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models, used in a distributed inference cluster comprising multiple physical nodes, each physical node being configured with at least one computing card and central processing unit memory, characterized in that, The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models includes: Step 1: In the background of normal model distributed inference, perform predictive memory cache scheduling asynchronously and periodically to generate and update the memory cache status table of the entire network's central processing unit in real time. Step 2: Construct a heterogeneous transmission cost matrix based on the network-wide central processing unit memory cache status table; Step 3: Obtain an integer linear programming model with group constraints. This model includes expert-computing card allocation variables, expert group-physical node binding variables, and the longest weight reception time variable across the entire network. It also includes constraints on the number of expert replicas, mandatory group affinity, memory and computational load, and the maximum time-consuming envelope constraint. Specifically, under the mandatory group affinity constraint, if a computing card loads an expert, the entire expert group to which that expert belongs is bound to the physical node where that computing card resides. Under the maximum time-consuming envelope constraint, the total weight reception time of any computing card does not exceed the longest weight reception time variable across the entire network. Step 4: With minimizing the longest weighted reception time across the entire network as the primary objective and minimizing the total global transmission volume as the secondary objective, perform bi-objective optimization on the integer linear programming model to obtain the optimal expert-computing card allocation scheme and expert group-physical node binding scheme; Step 5: Based on the optimal expert-computing card allocation scheme and the expert group-physical node binding scheme, each computing card pulls expert weights from the corresponding source location.

2. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 1, characterized in that, Step 1 includes: Step 11: The distributed inference cluster records the historical word processing volume of each expert in each layer in real time in the background through a sliding window, and uses the exponential moving average algorithm to calculate the expected load heat of each expert. When it is predicted that the total expert load of a certain computing card is about to exceed its computing power threshold, the expert with the fastest load growth on the computing card is marked as a high-probability migration candidate expert. Step 12: The distributed inference cluster calculates the node affinity score of each high-probability migration candidate expert to all surviving physical nodes in the cluster; Step 13: The distributed inference cluster selects the K target nodes with the highest node affinity scores for each high-probability migration candidate expert. During normal inference, the idle and fragmented bandwidth of the cross-node network is used to copy the weights of the candidate experts to the central processing unit memory of the K target nodes in advance and asynchronously. Step 14: The distributed inference cluster performs local mirroring and ring-shaped cross-node disaster recovery backup; Step 15: The distributed inference cluster generates and updates the network-wide central processing unit memory cache status table in real time, recording the storage location of all expert weights for prefetching, local mirroring, and ring backup.

3. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 2, characterized in that, The node affinity score in step 12 includes three dimensions: group constraint affinity, load complementarity, and network topology cost. Specifically, if the logical group to which the high-probability migration candidate expert belongs has been assigned to the target physical node, the score for the group constraint affinity dimension increases. If the total computing power load of the target physical node is currently decreasing and there is sufficient computing power margin, the score for the load complementarity dimension increases. If the underlying physical network bandwidth from the physical node currently holding the high-probability migration candidate expert to the target physical node is sufficient, the score for the network topology cost dimension increases.

4. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 2, characterized in that, Step 2 includes: Step 21: The distributed inference cluster scans in real time the status of all surviving computing cards in the current cluster, the central processing unit memory status of all physical nodes, and the global persistent storage status to obtain the current storage location of each expert; Step 22: Based on the expert storage location obtained in Step 21, and combined with the communication topology layer between the expert and the target computing card, set the transmission cost weight according to the principle of increasing cost, and calculate the initial transmission time baseline value for any expert to be transmitted to any surviving computing card. Step 23: Query the network-wide central processing unit memory cache status table. If the weight of the corresponding expert has been pre-fetched in the central processing unit memory of the physical node where the target computing card is located, then the initial transmission time baseline value of the expert to all computing cards in the physical node in Step 22 is reduced from the cross-node network transmission cost to the same-node high-speed serial computer expansion bus transmission cost to obtain the final heterogeneous transmission cost matrix.

5. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 4, characterized in that, The integer linear programming models with group constraints in step 3 include: Define the expert-computing card allocation variable as follows: , Indicates whether expert e is allocated to the video memory of computing card r; defines the expert group-physical node binding variable as... , Indicates whether expert group g is entirely bound to physical node n; defines the longest weighted reception time variable in the entire network as... , representing the maximum weight reception time among all computing cards; where e∈Expert set, r∈Computing card set, g∈Expert group set, n∈Physical node set; The expert replica number constraint is expressed by the formula. Obtain, among which The number of copies that expert group g needs to make; The forced group restriction affinity constraint is expressed by the formula Obtain, among which If r ∈ g and r ∈ n, ensure that if the computation card r loads expert e, then expert group g must be bound to node n; The constraints on video memory and computing load are expressed by the formula. And the computational load limit calculation based on the number of historical word units, where This is the maximum capacity of a single graphics card's video memory. The maximum time-consuming envelope constraint is expressed by the formula... Calculation and acquisition, ensuring that the total weight reception time for any computation card r does not exceed .

6. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 5, characterized in that, The objective function for the bi-objective optimization solution is: α is the weight coefficient corresponding to minimizing the longest weight reception time in the entire network, and β is the weight coefficient corresponding to minimizing the total global transmission volume. α > β, so as to prioritize suppressing the longest weight reception time in the entire network, and then penalize the total global transmission volume.

7. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 6, characterized in that, The method of setting transmission cost weights according to the principle of increasing cost includes: If the expert is already in the target computing card's video memory, the transfer cost weight is 0; If the expert uses the memory of other surviving computing cards on the same node, the first transmission cost weight corresponding to the NVLink interconnect is used; If the experts are in the same node's central processing unit memory, they should use the second transmission cost weight corresponding to the PCIe bus; If the expert uses the third transmission cost weight corresponding to the cross-node RDMA / IBGDA network in the video memory or central processing unit memory of other nodes; If the expert is lost globally, it needs to be loaded from the global persistent storage, using the corresponding fourth transmission cost weight for disk reading; the first transmission cost weight < the second transmission cost weight < the third transmission cost weight < the fourth transmission cost weight.

8. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 7, characterized in that, The distributed inference cluster performs local mirroring and ring-based cross-node disaster recovery backup, including: Local Mirror Backup: The expert weights held by each computing card in the current physical node are mirrored and backed up to the central processing unit memory of this node. In the event of a single card failure, other computing cards in the same node can pull the corresponding expert weights from the local central processing unit memory via the PCIe bus to take over. Ring-shaped cross-node disaster recovery backup: The physical nodes in the cluster are divided into a logical ring topology. By utilizing the idle network bandwidth during normal inference, the CPU memory of the Nth physical node asynchronously backs up the expert group weights of the N-1th physical node. In the event of a whole node failure, the Nth physical node can directly complete the expert takeover based on the backup in its local CPU memory without having to reload data from global persistent storage.

9. The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in claim 8, characterized in that, The heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models further includes: When the expert weight redistribution is triggered and the solution is completed, the CPU memory cache that has been hit is read into the corresponding computing card's video memory; the redundant prefetch cache that has not been hit is silently cleaned up by a background process with the lowest node affinity score as the priority elimination mechanism, or by the least recently used (LRU) mechanism, to release CPU memory space.

10. A heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models, characterized in that, The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models includes a distributed inference cluster with multiple physical nodes. Each physical node is configured with at least one computing card and a central processing unit memory. The heterogeneous cost-aware expert weight allocation system for ultra-large-scale hybrid expert models is used to obtain the heterogeneous cost-aware expert weight allocation method for ultra-large-scale hybrid expert models as described in any one of claims 1 to 9.

Images