Efficient reasoning and optimal energy consumption arrangement method based on MoE large model

Abstract

The invention relates to an efficient reasoning and optimal energy consumption arrangement method based on a MoE large model, and the method comprises the steps: constructing intra-layer semantic embedding and intra-cluster entropy measurement in a hierarchical semantic predictor, and forming a multi-stage routing probability; an interlayer coherent regular and sparse gate position is introduced into the gate position and threshold value module to obtain a controllable expert set and a soft clipping threshold value; a hierarchical cost function is designed in a metering and operation measurement module, end-to-end time delay is minimized, and bandwidth, energy consumption and routing stability are constrained; establishing a cross-node shaping and route merging mechanism in a traffic shaper, and aggregating high-similarity tokens to the same expert or the same rack expert copy in combination with an RDMA / NVLink topological channel; a verifiable execution log is provided. According to the method, on the premise of ensuring controllable model output quality, MoE reasoning can optimize time delay, bandwidth and energy consumption in a multi-node environment at the same time, and audit is realized through a verifiable execution log.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and distributed computing technology, specifically involving technologies such as online inference scheduling of hybrid expert large models, expert selection and pruning, cross-node communication orchestration, energy consumption-latency joint optimization and verifiable execution logs. Background Technology

[0002] Mixture-of-Experts (MoE) distributes input tokens to a small number of experts (typically the top-k) through a gating network to reduce computational cost, but it still faces the following challenges in large-scale distributed scenarios:

[0003] (1) Routing jitter and uneven load cause tail delay and cross-node communication amplification;

[0004] (2) Static Top-k is difficult to adapt to context changes, resulting in redundant expert calculations and energy waste;

[0005] (3) Cross-node fetching and parameter / activation synchronization cause bandwidth bottlenecks and queuing;

[0006] (4) There is a lack of unified modeling and verifiable measurement for multiple indicators (latency / bandwidth / energy consumption / output stability).

[0007] Existing solutions mostly perform local optimizations on single-layer routing or fixed pruning, and cannot achieve end-to-end joint orchestration and stable and auditable online inference pipelines. Summary of the Invention

[0008] Based on the aforementioned problems in the background technology, this invention aims to provide a method for efficient inference and energy-optimized orchestration of large hybrid expert (MoE) models based on hierarchical semantic routing and dynamic expert pruning. Specifically, under the premise of ensuring controllable model output quality, an end-to-end hierarchical semantic routing and dynamic pruning orchestration method is constructed, which enables MoE inference to simultaneously optimize latency, bandwidth and energy consumption in a multi-node environment, and achieves auditing through verifiable execution logs.

[0009] The technical solution adopted in this invention is as follows:

[0010] A method for efficient inference and energy-optimal orchestration based on MoE large models includes the following steps:

[0011] (1) In the hierarchical semantic predictor, construct the intra-layer semantic embedding and intra-cluster entropy measurement at the token level to form a multi-level routing probability.

[0012] (2) Introduce inter-layer coherence regularization and sparse gates in the gate and threshold modules to obtain a controllable Top-k expert set and soft pruning threshold.

[0013] (3) Design a hierarchical cost function in the measurement and operation module. Minimize end-to-end latency while constraining bandwidth, energy consumption, and routing stability;

[0014] (4) Establish a cross-node shaping and route merging mechanism in the traffic shaper, and combine RDMA / NVLink topology channels to aggregate highly similar tokens to the same expert or the same rack expert copy.

[0015] (5) Provide verifiable execution logs: perform commitment and sampling verification of routing decisions, pruning results, energy consumption metering and delay distribution.

[0016] Furthermore, in step (1), the multi-level routing probability The specific method is as follows:

[0017] right Perform intra-layer normalization and projection:

[0018] For the Layer token representation Clustering and temperature calibration are performed to obtain the usage probability of expert e:

[0019] Furthermore, in step (2), the specific method for introducing inter-layer coherence regularization is as follows:

[0020] Furthermore, in step (2), the specific methods for sparse gate positions and soft clipping are as follows:

[0021] Let the Top-k set be... and with threshold Do soft cutting:

[0022] dynamic Where H eff It is effective entropy;

[0023] threshold Through dynamic KKT conditions are updated online:

[0024] Furthermore, in step (2), the method for ensuring the consistency of the trimming and the robustness of the output is as follows:

[0025] For the experts who were cut Correction using distillation: Approximate compensation for the pruned contribution without introducing forward computation;

[0026] For high-entropy, low-probability experts, a fallback merge is enabled: if the batch entropy... If the threshold is exceeded, the output is quantized and weighted and then merged.

[0027] Furthermore, in step (3), the specific method for minimizing the combined costs of latency, bandwidth, energy consumption, and stability is as follows:

[0028]

[0029] Q uv For the byte traffic of link u→v, β uv The cost is per byte; TV is total variation.

[0030] Furthermore, in step (3), the method for establishing a unified model of latency, bandwidth, energy consumption, and routing stability is as follows:

[0031] Single-layer delay:

[0032] Energy consumption:

[0033] Router "drift" penalty:

[0034] Furthermore, in step (4), the specific method for cross-node reshaping and route merging is as follows:

[0035] Based on similarity kernel Merge tokens from the same cluster into batches and map them onto the topological graph G = (V, E) to find the shortest set of expert replicas. Solving for minimum cost flow: Where c bv By link cost α uv +β uv s b The replica load is given;

[0036] Consistent hashing and replica affinity constraints are used for replicas on different racks to reduce migration jitter.

[0037] Furthermore, in step (5), the method for generating the execution log can be verified as follows: a log is generated for each batch of routing matrix, pruning threshold, minimum cost flow allocation, and latency and energy consumption metering. i Calculate root = Merkle(log1,...); Random sampling index set Verification is performed to ensure that logs are non-repudiable and traceable across batches.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention provides an integrated method for hierarchical semantic routing, dynamic expert pruning, cross-node traffic shaping, and energy-optimal orchestration for large MoE models. On a 16×GPU+IB cluster, compared to the static Top-2 MoE routing baseline, throughput is increased by 1.7×–2.2×, p95 latency is reduced by 25%–43%, energy consumption per unit sample is reduced by 28%–40%, and output consistency is maintained within ±0.2%. Detailed Implementation

[0040] I. Parameter Description of the Invention

[0041] 1. Model and routing related parameters

[0042] Layer index; e: expert index; t: time / step index;

[0043] No. Layer token representation; φ(·): projection mapping;

[0044] Experts use probability; The probability after cropping;

[0045] temperature; Routing weight;

[0046] Top-k size; Soft clipping threshold;

[0047] Intra-layer token cluster set; κ(·,·): similarity kernel;

[0048] Top-k expert group; H eff Effective entropy;

[0049] Interlayer coherence regularization; Routing loss.

[0050] 2. Topology and Shaping Related Parameters

[0051] G = (V, E): computation / communication topology; u, v ∈ V: nodes;

[0052] α uv Fixed latency (startup overhead); β uv Cost per byte;

[0053] Quv : Link u→v traffic; C v : Copy capacity;

[0054] x b,v ∈{0,1}: the allocation variable from batch b to copy v; s b Batch size;

[0055] c bv Cost in the minimum cost flow; A collection of copies of expert e.

[0056] 3. Standardize cost and indicator-related parameters Total cost; λ,μ,η: trade-off coefficients (bandwidth / energy consumption / drift);

[0057] No. Layer expected delay; L queue ,L comp ,L comm Queuing / computation / communication delays;

[0058] Energy consumption; p base ,p act Baseline / Active Power Consumption Parameters;

[0059] TV(·,·): Totally varied distance; ω tenant Tenant weight;

[0060] The merged output is compared with the original output; Output differences.

[0061] 4. Log and verification related parameters

[0062] log i : Execution record of the i-th batch; root: Merkle root;

[0063] Sampling index set; Verify(·): Replay verification function. II. Specific Implementation Methods

[0065] A method for efficient inference and energy-optimal orchestration based on MoE large models includes the following steps:

[0066] 1. In the hierarchical semantic predictor, construct intra-layer semantic embedding and intra-cluster entropy measurement at the token level to form multi-level routing probabilities.

[0067] Multi-level routing probability The specific method is as follows:

[0068] right Perform intra-layer normalization and projection:

[0069] For the Layer token representation Clustering and temperature calibration are performed to obtain the usage probability of expert e:

[0070] 2. By introducing inter-layer coherence regularization and sparse gates into the gate and threshold modules, a controllable Top-k expert set and soft pruning threshold are obtained.

[0071] The specific method for introducing inter-layer coherence regularization is as follows: Suppress inter-layer routing jitter and improve cacheability and batch merging efficiency.

[0072] The specific methods for sparse door positions and soft clipping are as follows:

[0073] Let the Top-k set be... and with threshold Do soft cutting:

[0074] dynamic Where H eff It is effective entropy;

[0075] threshold Through dynamic KKT conditions are updated online:

[0076] The method for ensuring the consistency of the trimming and the robustness of the output is as follows:

[0077] For the experts who were cut Correction using distillation: Approximate compensation for the pruned contribution without introducing forward computation;

[0078] For high-entropy, low-probability experts, a fallback merge is enabled: if the batch entropy... If the threshold is exceeded, the output is quantized and weighted and then merged.

[0079] 3. Design a hierarchical cost function in the measurement and operation module. Minimize end-to-end latency while constraining bandwidth, energy consumption, and routing stability.

[0080] The specific method for minimizing the combined costs of latency, bandwidth, energy consumption, and stability is as follows:

[0081]

[0082] Q uv For the byte traffic of link u→v, β uv The cost is per byte; TV is total variation.

[0083] The unified model for latency, bandwidth, energy consumption, and routing stability is established as follows:

[0084] Single-layer delay:

[0085] Energy consumption:

[0086] Router "drift" penalty:

[0087] 4. Establish a cross-node shaping and route merging mechanism in the traffic shaper, and combine RDMA / NVLink topology channels to aggregate highly similar tokens to the same expert or the same rack expert copy.

[0088] The specific methods for cross-node reshaping and route merging are as follows:

[0089] Based on similarity kernel Merge tokens from the same cluster into batches and map them onto the topological graph G = (V, E) to find the shortest set of expert replicas. Solving for minimum cost flow: Where c bv By link cost α uv +β uv S b The replica load is given;

[0090] Consistent hashing and replica affinity constraints are used for replicas on different racks to reduce migration jitter.

[0091] 5. Provide verifiable execution logs: perform commitment and sampling verification of routing decisions, pruning results, energy consumption metering, and delay distribution.

[0092] The method for proposing verifiable execution logs is as follows: For each batch, a log is generated containing the routing matrix, pruning threshold, minimum cost flow allocation, latency, and energy consumption metering. i Calculate root = Merkle(log1,...); Random sampling index set Perform verification; if the verification passes, archive / statistics are generated; if the verification fails, an alarm / rollback / correction is triggered to ensure that the logs are non-repudiable and traceable across batches.

Claims

1. A method for efficient inference and energy-optimal orchestration based on MoE large models, characterized in that, Includes the following steps: (1) In the hierarchical semantic predictor, construct the intra-layer semantic embedding and intra-cluster entropy measurement at the token level to form a multi-level routing probability. (2) Introduce inter-layer coherence regularization and sparse gates in the gate and threshold modules to obtain a controllable Top-k expert set and soft pruning threshold. (3) Design a hierarchical cost function in the measurement and operation module. Minimize end-to-end latency while constraining bandwidth, energy consumption, and routing stability; (4) Establish a cross-node shaping and route merging mechanism in the traffic shaper, and combine RDMA / NVLink topology channels to aggregate highly similar tokens to the same expert or the same rack expert copy. (5) Provide verifiable execution logs: perform commitment and sampling verification of routing decisions, pruning results, energy consumption metering and delay distribution.

2. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (1), the multi-level routing probability is calculated. The specific method is as follows: right Perform intra-layer normalization and projection: φ(z)=W l LN(z); Representation of the l-th layer token Clustering and temperature calibration are performed to obtain the usage probability of expert e:

3. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (2), the specific method for introducing inter-layer coherence regularization is as follows:

4. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (2), the specific methods for sparse gate positions and soft clipping are as follows: Let the Top-k set be... and with threshold θ e Do soft cutting: Dynamic Top-k l : Where H eff It is effective entropy; threshold θ l Through dynamic Top-k l KKT conditions are updated online:

5. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (2), the method for controlling the consistency of the trimming and the robustness of the output is as follows: For the experts who were cut Correction using distillation: Approximate compensation for the pruned contribution without introducing forward computation; For high-entropy, low-probability experts, a fallback merge is used: if the batch entropy H(π) l If the value exceeds the threshold, the output will be quantized and weighted and then merged.

6. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (3), the specific method for minimizing the combined costs of latency, bandwidth, energy consumption, and stability is as follows: Q uv For the byte traffic of link u→v, β uv The cost is per byte; TV is total variation.

7. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (3), the method for establishing a unified model of latency, bandwidth, energy consumption, and routing stability is as follows: Single-layer delay: Energy consumption: P l,e =p base +p act ·f(batch,d); Router "drift" penalty:

8. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (4), the specific method for cross-node reshaping and route merging is as follows: Based on similarity kernel Merge tokens from the same cluster into batches and map them onto the topological graph G = (V, E) to find the shortest set of expert replicas. Solving for minimum cost flow: Where c bv By link cost α uv +β uv s b The replica load is given; Consistent hashing and replica affinity constraints are used for replicas on different racks to reduce migration jitter.

9. The method for efficient inference and energy-optimal orchestration based on MoE large models according to claim 1, characterized in that, In step (5), the verifiable method for generating the execution log is as follows: a log is generated for each batch, including the routing matrix, pruning threshold, minimum cost flow allocation, and latency and energy consumption metering. i Calculate root = Merkle(log1,...); Random sampling index set Verification is performed to ensure that logs are non-repudiable and traceable across batches.

Images