A wide-area network-oriented hybrid expert model cross-node dynamic routing method and system
By sharding the MoE model by layer and expert two-dimensional granularity, and combining it with a dual-mode routing mechanism and dynamic replica scheduling, the efficiency and stability issues of hybrid expert model deployment and inference in a wide area network environment are solved, achieving efficient resource utilization and latency reduction.
Patent Information
- Application Number
- CN202610772885.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-01
- Publication Date
- 2026-08-25
AI Technical Summary
Existing technologies struggle to efficiently and stably deploy and infer hybrid expert models in wide area network environments, especially on consumer-grade GPU nodes, resulting in low resource utilization and high latency.
The method and system of cross-node dynamic routing using the hybrid expert model for wide area networks is adopted. By dividing the MoE model into layers and expert two-dimensional granularity, it is distributed and deployed on multiple GPU nodes. A dual-mode routing mechanism is adopted, including Router-First Routing and predictive routing. Combined with the activation of the predictive model and dynamic replica scheduling, efficient and stable operation is ensured.
It improves resource utilization, significantly reduces inference latency, ensures the efficient and stable operation of the MoE model in a wide area network environment, and has a fast failure rollback mechanism to prevent request failure.
Smart Images
Figure CN122640339A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a hybrid expert model cross-node dynamic routing method and system for wide area networks. Background Technology
[0002] Hybrid expert (MoE) models have become one of the mainstream architectures in the field of large language models in recent years. Representative models include Mixtral, DeepSeek-V3 (671B total parameters / 37B activation), and Qwen3-235B-A22B (235B total parameters / 22B activation).
[0003] The core feature of MoE is that the model parameters are split into several experts, each of which is an independent feedforward network. The feed-forward part in each transformer block is replaced by a router (gate network) + N experts.
[0004] Typical reasoning process of MoE:
[0005] (1) Input hiddenstate is processed by the router;
[0006] (2) The router outputs an N-dimensional probability distribution and selects the top-K experts;
[0007] (3) The K activated experts each process the hidden state, and the weighted sum is used to obtain the final output;
[0008] (4) K is usually much smaller than N (e.g., K=2 / N=8, or K=8 / N=256), i.e. sparse activation. Summary of the Invention
[0009] The purpose of this invention is to address the problems existing in the background technology by proposing a hybrid expert model cross-node dynamic routing method and system for deploying and inferring the MoE model on consumer-grade GPU nodes in wide area networks.
[0010] The technical solution of this invention: A hybrid expert model cross-node dynamic routing method and system for wide area networks, comprising the following steps:
[0011] S1. Divide the MoE model into two-dimensional granularities (layer, expert) and distribute them across several GPU nodes; each node loads several (layer, expert) combinations; the router sublayer has redundant copies on multiple nodes;
[0012] S2. First-stage routing (Router-FirstRouting): After the request arrives at the node that loads the router, it first executes the router output to activate the expert set, then distributes the hidden state to the node that loads the activated experts, aggregates the outputs of each expert, and then enters the next layer;
[0013] S3. Collect historical records (prompt features, layer, actual activation experts) and periodically train the activation prediction model;
[0014] S4. Second-stage routing (predictive routing): When the Gateway receives a request, it calls the prediction model to directly predict the set of active experts at each layer, constructs a complete path based on the prediction, and inserts it into the request header; the request is executed according to the predicted path, and if the actual activation does not match the prediction, it is forwarded to the correct node in real time.
[0015] The number of S5.expert replicas is dynamically adjusted based on the real-time activation frequency; experts with high activation rates automatically receive more replicas.
[0016] S6. When the prediction error rate exceeds the threshold, automatically fall back to the first stage Router-FirstRouting as a fallback.
[0017] Preferably, the two-dimensional deployment at the expert granularity also includes fault domain distribution constraints for (layer, expert) combinations to ensure that multiple copies of the same (layer, expert) are distributed across different ISP / power grid regions.
[0018] Preferably, the activation prediction model adopts a small transformer architecture with approximately 50M parameters and an inference latency of <5ms.
[0019] Preferably, the prediction model is activated for offline training once a week, with daily incremental fine-tuning.
[0020] Preferably, the execution of the predicted route includes (a) prediction, (b) path construction, (c) verification (comparing the output of the node-side router with the prediction), and (d) real-time forwarding when errors occur.
[0021] Preferably, the expert replica scheduling refers to the layer-granular dynamic replication scheduling method of patent P-002.
[0022] Preferably, the rollback determination uses the accuracy of a sliding window; if the accuracy is less than 50% for 5 minutes, a rollback is triggered.
[0023] A hybrid expert model (MoE) cross-node dynamic routing system for wide area networks includes a sharding deployment module, a dual-mode routing scheduling module, an activation prediction training module, a path verification and forwarding module, an expert replica dynamic optimization module, and a routing strategy fallback switching module. The sharding deployment module completes two-dimensional sharding deployment and fault domain isolation at the (layer, expert) level. The dual-mode routing scheduling module coordinates the collaborative operation of the first-stage native routing and the second-stage predicted routing. The activation prediction training module completes historical data collection and model iterative optimization. The path verification and forwarding module implements route prediction verification and abnormal forwarding. The expert replica dynamic optimization module adaptively expands replicas based on activation frequency. The routing strategy fallback switching module automatically switches routing modes based on sliding window accuracy metrics. The overall architecture is adaptable to wide area network scenarios with multiple heterogeneous GPU nodes, ensuring efficient and stable operation of the MoE model across regions.
[0024] Compared with the prior art, the present invention has the following beneficial technical effects:
[0025] In this invention, the (layer, expert) two-dimensional granularity is used as the basic unit for MoE deployment. Compared with the whole model as a single machine deployment or the layer-based fragmentation of academic prototypes, the granularity is finer and the resource utilization is higher.
[0026] MoE routing evolves from accurate but slow to fast but with fallback mechanisms. The Router-First phase ensures reliability, and as data accumulates, a training prediction model is used to gradually replace the old model. This evolution path is independent of specific models and can be applied to any large MoE model.
[0027] The prediction model is trained directly on path pre-construction tasks and is seamlessly integrated with the Loom system's InstantRoute mechanism.
[0028] This invention designs a three-level fallback: real-time forwarding at the node local level → cross-node query → fallback to Router-First, ensuring that no prediction error will cause the request to fail. Attached Figure Description
[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a schematic diagram of the model architecture in this invention;
[0031] Figure 2This is a schematic diagram of the two-dimensional deployment in this invention. Detailed Implementation
[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0033] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0034] Example
[0035] like Figure 1-2 As shown, the present invention proposes a hybrid expert model cross-node dynamic routing method for wide area networks, which includes the following steps:
[0036] S1. Divide the MoE model into two-dimensional granularities (layer, expert) and distribute them across several GPU nodes; each node loads several (layer, expert) combinations; the router sublayer has redundant copies on multiple nodes;
[0037] S2. First-stage routing (Router-FirstRouting): After the request arrives at the node that loads the router, it first executes the router output to activate the expert set, then distributes the hidden state to the node that loads the activated experts, aggregates the outputs of each expert, and then enters the next layer;
[0038] S3. Collect historical records (prompt features, layer, actual activation experts) and periodically train the activation prediction model;
[0039] S4. Second-stage routing (predictive routing): When the Gateway receives a request, it calls the prediction model to directly predict the set of active experts at each layer, constructs a complete path based on the prediction, and inserts it into the request header; the request is executed according to the predicted path, and if the actual activation does not match the prediction, it is forwarded to the correct node in real time.
[0040] The number of S5.expert replicas is dynamically adjusted based on the real-time activation frequency; experts with high activation rates automatically receive more replicas.
[0041] S6. When the prediction error rate exceeds the threshold, automatically fall back to the first stage Router-FirstRouting as a fallback.
[0042] The two-dimensional deployment at the expert granularity also includes fault domain distribution constraints for (layer, expert) combinations, ensuring that multiple copies of the same (layer, expert) are distributed across different ISP / power grid regions.
[0043] The activation prediction model employs a small transformer architecture with approximately 50M parameters and an inference latency of <5ms.
[0044] The prediction model is activated and trained offline once a week, with daily incremental fine-tuning.
[0045] The execution of predictive routing includes (a) prediction, (b) path construction, (c) verification (comparing the output of the node-side router with the prediction), and (d) real-time forwarding in case of errors.
[0046] Expert replica scheduling references the layer-level dynamic replicability scheduling method of patent P-002.
[0047] Rollback is determined using the accuracy of a sliding window; if the accuracy is less than 50% for 5 minutes, a rollback is triggered.
[0048] A hybrid expert model (MoE) cross-node dynamic routing system for wide area networks includes a sharding deployment module, a dual-mode routing scheduling module, an activation prediction training module, a path verification and forwarding module, an expert replica dynamic optimization module, and a routing strategy fallback switching module. The sharding deployment module completes two-dimensional sharding deployment and fault domain isolation at the (layer, expert) level. The dual-mode routing scheduling module coordinates the collaborative operation of the first-stage native routing and the second-stage predicted routing. The activation prediction training module completes historical data collection and model iterative optimization. The path verification and forwarding module implements route prediction verification and abnormal forwarding. The expert replica dynamic optimization module adaptively expands replicas based on activation frequency. The routing strategy fallback switching module automatically switches routing modes based on sliding window accuracy metrics. The overall architecture is adaptable to wide area network scenarios with multiple heterogeneous GPU nodes, ensuring efficient and stable operation of the MoE model across regions.
[0049] Phase 1 (M6): Router-FirstRouting. Requests first go to the node that loads the router, where the router outputs an active set, which is then distributed to the nodes that load the active sets.
[0050] Phase Two (M7+): Predictive Routing. The Gateway trains a prediction model based on historical requests to directly predict the activation set and construct the complete path in advance, avoiding routing through router nodes.
[0051] Both phases share the underlying infrastructure: expert-level deployment, dynamic replica scheduling, and cross-node QUIC communication. In case of prediction errors, a fallback to phase one is implemented as a fallback.
[0052] Distributed deployment at the expert level
[0053] Deploy the MoE model at a two-dimensional granularity of (layer_id, expert_id):
[0054] Each (layer, expert) is an independent deployment unit.
[0055] Each node loads several combinations of (layer, expert)
[0056] The router sublayer (the gate network for each transformer block) is deployed separately on the router node.
[0057] The router sublayer has a small weight (only N-dimensional output), which allows for redundancy across multiple nodes.
[0058] Deployment example (assuming 256 experts / layer):
[0059] Node A (RTX5090 32GB):
[0060] -layer0router
[0061] -layer0expert{3, 5, 7, 17, 22, 50, 102}
[0062] -layer1router
[0063] -layer1expert{1, 9, 12, 50, 88, 200}
[0064] Node B (RTX5090 32GB):
[0065] -layer0expert{1, 17, 22, 50, 88, 100, 200, 250}
[0066] -layer1expert{3, 9, 22, 100, 102, 200, 250}
[0067] Node C (RTX4090 24GB):
[0068] -layer0expert{7, 12, 22, 50, 88, 100} / / Partial redundancy
[0069] -layer2router
[0070] -...
[0071] Key observation: The expert22 (high frequency) is installed in A / B / C, while the expert250 (low frequency) is only installed in B (low redundancy).
[0072] Router-FirstRouting (Phase 1)
[0073] Router-FirstRouting is the basic routing mechanism of this invention, used in the M6 phase. The processing flow is as follows: Figure 1 As shown: Advantages of uter-FirstRouting:
[0074] The routing decision is accurate (based on the actual router output);
[0075] It does not rely on any prediction model and is simple and reliable to implement;
[0076] Disadvantages of Router-FirstRouting:
[0077] Each layer requires two cross-node jumps: router node → expert node → back to router node (if the expert node is on another node), introducing twice the RTT.
[0078] Total latency = N layers × (router processing + 2 × cross-node RTT), N = 36 layers + 50ms RTT, latency = 36 × 100ms = 3.6 seconds, unacceptable.
[0079] Predictive Routing (Phase Two)
[0080] Predictive routing is an advanced routing mechanism of this invention, enabled in the M7+ phase. The core idea is to train a small predictive model based on historical request data, directly predicting the set of experts activated at each layer at the Gateway end, constructing the complete path in advance, and avoiding the overhead of intermediate router nodes in Router-First.
[0081] Predictive model design
[0082] Inputs and outputs of the prediction model:
[0083] Input features:
[0084] -prompt's embedding (takes the first sentence / summary)
[0085] - User ID (optional, reflects user behavior patterns)
[0086] - Model ID (Supports multi-model fusion prediction)
[0087] - Context length
[0088] -Historical conversation topics (optional)
[0089] Output:
[0090] - Set of predicted activation experts for each layer (L=0 to L_max):
[0091] layer0: [3, 22, 50] (confidence level 0.85)
[0092] layer1: [1, 9, 50] (confidence level 0.78) ...
[0094] Model architecture: Small transformer (approximately 50M parameters)
[0095] Inference latency: <5ms
[0096] Training the prediction model:
[0097] (1) Data source: Historical records of all Router-First routes in the M5 / M6 phase, recording each (prompt, layer, actual active expert set) triplet;
[0098] (2) Training objective: Minimize the symmetric difference between the predicted activation set and the actual activation set;
[0099] (3) Training frequency: Offline training once a week, with daily incremental fine-tuning;
[0100] (4) Deployment location: Embedded within Gateway (local cache model copy for each Gateway instance);
[0101] Prediction accuracy and delay benefits
[0102] Scene Prediction accuracy Average additional expenses 100% accurate ideal 0 hops (completely skips router nodes) 80% accurate Reality 0.2 jumps / floor (20% chance of going all the way to the bottom) 60% accurate barely satisfactory 0.4 jumps per floor (40% chance of going to the bottom) Router-First (baseline) not applicable 1 jump / floor (with a transfer on each floor)
[0103] Even with a prediction accuracy of only 60% (barely satisfactory), the average additional cost is only 40% of that of Router-First, with significantly reduced latency. The target accuracy is 80%+.
[0104] Dynamic replica scheduling at the expert level
[0105] The expert replica scheduling of this invention is a specialized application of dynamic replication scheduling in the MoE scenario. Specific rules:
[0106] High-frequency expert (activation rate > 5%): Number of replicas = ceil(QPS × activation rate / single node capacity), typically 5-10 replicas.
[0107] Mid-frequency expert (activation rate 1-5%): 2-4 copies
[0108] Low-frequency expert (activation rate <1%): Maintenance tolerance lower limit R_min = 2
[0109] Router sublayer: Since all requests must pass through it, the number of replicas = ceil(number of cluster nodes × 0.3)
[0110] Replica scheduling is handled by MetaScheduler (described in another patent in this series, P-002).
[0111] Error rollback mechanism
[0112] Fallback when route prediction fails:
[0113] (1) Scenario A: The predicted activation expert is not loaded on any path node;
[0114] (2) Processing: After the request arrives at the node, the actual router outputs the active set, queries the LongTermRoute for the node that loads the expert, and forwards it in real time (adding 1 hop);
[0115] (3) Scenario B: A node on the predicted path is offline;
[0116] (4) Processing: Reuse the InstantRoutebackup mechanism of the P-001 patent for local switching;
[0117] (5) Scenario C: Prediction confidence is too low (<0.5);
[0118] (6) Handling: Directly return to Router-FirstRouting and skip the prediction.
[0119] Using the (layer, expert) two-dimensional granularity as the basic unit for MoE deployment, compared with the whole model as a single machine deployment or the layer-based fragmentation of academic prototypes, the granularity is finer and the resource utilization is higher.
[0120] MoE routing evolves from accurate but slow to fast but with fallback mechanisms. The Router-First phase ensures reliability, and as data accumulates, a training prediction model is used to gradually replace the old model. This evolution path is independent of specific models and can be applied to any large MoE model.
[0121] The prediction model is trained directly on path pre-construction tasks and is seamlessly integrated with the Loom system's InstantRoute mechanism.
[0122] This invention designs a three-level fallback: real-time forwarding at the node local level → cross-node query → fallback to Router-First, ensuring that no prediction error will cause the request to fail.
[0123] Example 1: Deployment of DeepSeek-V3 on 3 5090 frames
[0124] Scenario: DeepSeek-V3 model (671B total parameters, 256 experts / layer, 8 / 256 activations) deployed on 3 RTX5090 nodes (each with 32GB VRAM).
[0125] Deployment plan:
[0126] VRAM usage for node loading content
[0127] Node A contains all layer routers and high-frequency experts (approximately 80 each), totaling approximately 28GB.
[0128] Node B mid-frequency experts (approximately 120 each, overlapping with part of A) 30GB
[0129] Node C low-frequency expert + fault-tolerant redundancy of approximately 30GB
[0130] The total number of parameters is 671B, but each node only loads about 30B of parameters (15% of the total parameters). Through dynamic replication and fault domain design, the cluster can still work even if any node goes offline.
[0131] Example 2: Execution of a typical chat request (Router-First phase)
[0132] Customer Question: How many solar terms are there in traditional Chinese culture?
[0133] (1) Upon receiving the request, the Gateway constructs an InstantRoute = [Node A (layer0router + partialexpert), Node B (remainingexpert), ...]
[0134] (2) Request to reach node A, execute layer0router → output activation [12, 30, 50, 80, 100, 150, 200, 220]
[0135] (3) Node A has expert{3, 5, 7, 17, 22, 50, 102, ...} (where 50 is active).
[0136] (4) Node A executes expert50, and the remaining 7 experts are sent to the node that loaded them via QUIC.
[0137] (5) All expert nodes send the results back to node A.
[0138] (6) Node A performs a weighted summation to obtain the layer0 output, which is then forwarded to the layer1 router node.
[0139] (7) Repeat the process from layer 1 to layer 60.
[0140] (8) The last layer completes the sample, and the token flows back to the Gateway.
[0141] Estimate end-to-end delay:
[0142] The average latency per layer is 60ms (5ms for router + 30ms for expertforward + 25ms for cross-node communication).
[0143] DeepSeek-V3 has 60 layers, with a single token latency of approximately 3.6 seconds.
[0144] The client displays the information in a streaming manner, and users perceive it to take approximately 5-10 seconds to answer a complete question.
[0145] Example 3: Predictive Routing Phase (M7+ Enabled)
[0146] For the same request, predictive routing is enabled in phase M7:
[0147] (1) Gateway calls the local prediction model (<5ms) and directly outputs the prediction activation set: layer0: [12, 30, 50, 80, 100, 150, 200, 220]
[0148] (2) layer1: [...]
[0149] (3) The Gateway constructs a complete InstantRoute, which directly routes to the node that loads the prediction expert;
[0150] (4) The request arrives at node A, and the router is run to verify the prediction → the actual activation is consistent with the prediction;
[0151] (5) Execute the local expert directly and forward it to the next layer (no router required);
[0152] (6) If the actual activation is inconsistent with the prediction (20% probability), forward it to the correct node in real time;
[0153] Estimate end-to-end delay:
[0154] The average time per layer is 25ms (if the prediction is correct, directly use expertforward + forwarding);
[0155] The latency for 60 layers is approximately 1.5 seconds, a 60% reduction compared to Router-First.
[0156] The total latency was reduced from 5-10 seconds to 2-4 seconds, which is close to 2-3 times that of a standalone solution.
[0157] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.
Claims
1. A hybrid expert model cross-node dynamic routing method for wide area networks, characterized in that: Includes the following steps: S1. Divide the MoE model into (layer, expert) two-dimensional granularity pieces and distribute them across several GPU nodes; each node loads several (layer, expert) combinations; the router sublayer has redundant copies on multiple nodes; S2. First-stage routing (Router-FirstRouting): After the request arrives at the node that loads the router, it first executes the router output to activate the expert set, then distributes the hidden state to the node that loads the activated experts, aggregates the outputs of each expert, and then enters the next layer; S3. Collect historical records (prompt features, layer, actual activation experts) and periodically train the activation prediction model; S4. Second-stage routing (predictive routing): When the Gateway receives a request, it calls the prediction model to directly predict the set of active experts at each layer, constructs a complete path based on the prediction, and inserts it into the request header; the request is executed according to the predicted path, and if the actual activation does not match the prediction, it is forwarded to the correct node in real time. The number of S5.expert replicas is dynamically adjusted based on the real-time activation frequency; experts with high activation rates automatically receive more replicas. S6. When the prediction error rate exceeds the threshold, automatically fall back to the first stage Router-FirstRouting as a fallback.
2. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 1, characterized in that, The two-dimensional deployment at the expert granularity also includes fault domain distribution constraints for (layer, expert) combinations, ensuring that multiple copies of the same (layer, expert) are distributed across different ISP / power grid regions.
3. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 2, characterized in that, The activation prediction model employs a small transformer architecture with approximately 50M parameters and an inference latency of <5ms.
4. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 3, characterized in that, The prediction model is activated and trained offline once a week, with daily incremental fine-tuning.
5. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 4, characterized in that, The execution of predictive routing includes (a) prediction, (b) path construction, (c) verification (comparing the output of the node-side router with the prediction), and (d) real-time forwarding in case of errors.
6. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 5, characterized in that, Expert replica scheduling references the layer-level dynamic replicability scheduling method of patent P-002.
7. The hybrid expert model cross-node dynamic routing method for wide area networks according to claim 6, characterized in that, Rollback is determined using the accuracy of a sliding window; if the accuracy is less than 50% for 5 minutes, a rollback is triggered.
8. A hybrid expert model cross-node dynamic routing system for wide area networks (WANs), based on the hybrid expert model cross-node dynamic routing method for WANs according to claim 7, characterized in that, The system includes a sharding deployment module, a dual-mode routing scheduling module, an activation prediction training module, a path verification and forwarding module, an expert replica dynamic optimization module, and a routing strategy fallback switching module. The sharding deployment module completes two-dimensional granular sharding deployment and fault domain isolation deployment (layer, expert). The dual-mode routing scheduling module coordinates the collaborative operation of the first-stage native routing and the second-stage predicted routing. The activation prediction training module completes historical data collection and model iterative optimization. The path verification and forwarding module realizes route prediction verification and abnormal forwarding. The expert replica dynamic optimization module adaptively expands replicas based on activation frequency. The routing strategy fallback switching module completes automatic fallback switching of routing modes based on the sliding window accuracy index. The overall architecture is adapted to wide area network scenarios with multiple heterogeneous GPU nodes, ensuring efficient and stable operation of the MoE model for cross-regional inference.