A multi-core particle reasoning acceleration method, system and application for hybrid expert model

Abstract

The application discloses a kind of mixed expert model-oriented multi-core particle reasoning acceleration method, comprising: for input token sequence, in current MoE layer, according to the target expert selected by input token for the gating network, construct token distribution mapping to target core particle;Token embedding corresponding to each token is broadcasted and distributed along the ring interconnection topology, so that target core particle receives and caches the token embedding corresponding to local expert;Each target core particle is based on cached token embedding and the expert weight stored in the core particle, the target expert hitting the storage range of the core particle is executed calculation, and the sum corresponding to token is obtained;The sum corresponding to the same token is aggregated along the ring interconnection topology, and the layer output embedding of current MoE layer is obtained;The load of each computing core particle is monitored, and the distribution mapping of next MoE layer is dynamically adjusted according to load statistical information.The application also discloses a system for implementing the above method and application, with wide application scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence model inference acceleration and chip computing architecture technology, and in particular to a multi-chip inference acceleration method, system and application for hybrid expert models. Background Technology

[0002] Hybrid Expert (MoE) architecture achieves an effective balance between capacity expansion and computational cost for large language models through a sparse activation mechanism, and has become the common technical choice for mainstream large language models (including DeepSeek-V3 / DeepSeek-R1, Qwen2-MoE, etc.). Taking DeepSeek-V3 as an example, each MoE layer contains 1 shared expert and 256 routing experts. Each token selects 8 routing experts, routing to a maximum of 4 nodes. This sparse activation design allows the model to have a large capacity without significantly increasing computational cost.

[0003] However, the widespread deployment of the MoE architecture also presents significant challenges at the hardware level. First, Expert Parallelism (EP) requires all-to-all communication at each MoE layer, distributing and merging tokens across devices. When the number of experts is large (e.g., DeepSeek-V3's 256 routing experts) and EPs span multiple nodes, inter-device communication bandwidth becomes a critical bottleneck, severely limiting inference throughput. Second, sparse activation causes significant load unevenness—the activation frequency of each expert varies greatly under different inputs, leading to some computational cores being overloaded while others remain idle. Real-world testing and engineering experience with DeepSeek-R1 show that expert routing and cross-device communication have become critical bottlenecks. Finally, multiple experts are distributed across different chips or cores, and each time an expert is hit, only a very small portion of all computing resources is activated, resulting in a significant waste of computing power.

[0004] To address the hardware deployment challenges of the aforementioned MoE architecture, existing research has attempted to find solutions from multiple perspectives. Regarding expert distribution strategies, recent research has proposed Full Sharded Expert Data-parallelism (FSE-DP), which enables adaptive computational communication overlap and load balancing on multi-core accelerators, achieving a 1.22 to 2.00x speedup compared to mainstream baselines and saving up to 78.8% of on-chip storage resources. In terms of parallel scheduling, FinDEP, through a fine-grained task scheduling algorithm, improves inference throughput to 1.61 times that of previous methods on DeepSeek-V2 and Qwen3-MoE models, achieving a maximum speedup of 1.24x on 32-GPU systems; MoEntwine proposes Entwined Ring Mapping (ER-Mapping), achieving up to 62% reduction in communication throughput through communication link balancing. At the chip level, the maturity of Chiplet technology provides a new hardware foundation for MoE inference acceleration. The current mainstream Chiplet interconnect standard UCIe 3.0 supports data rates of 48 GT / s and 64 GT / s, enabling high-bandwidth, low-latency inter-chip connections within the package.

[0005] While the aforementioned solutions optimize MoE inference at different levels, they all share a common problem: a lack of systematic design for an expert data sharing mechanism in chiplet / multi-chip systems. In existing expert parallel schemes, each token, after routing decisions, needs to be transmitted all-to-all, first to the chiplet storing the corresponding expert before computation. During this process, there is a significant amount of redundant access to the input data processed by each chiplet; expert parameters are also stored independently across chipslet, making reuse difficult, resulting in wasted storage resources and DRAM bandwidth pressure. Therefore, there is an urgent need to design a cross-chiplet sharing mechanism for input data and expert parameters in multi-chip / multi-chiplet MoE inference architectures. This mechanism should fully utilize the high bandwidth interconnects of chiplets while effectively reducing the frequency of off-chip memory access and the amount of on-chip storage resources used, achieving high-throughput inference under uniform load. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method, system, and application for accelerating multi-core inference for hybrid expert models.

[0007] In a first aspect, the present invention provides a multi-core inference acceleration method for hybrid expert models. The method is based on an accelerator array containing M computational cores, wherein the M computational cores are sequentially connected end-to-end through interconnect interfaces to form a ring interconnect topology, including:

[0008] For the input token sequence, in the current MoE layer, a distribution mapping from token to target kernel is constructed based on the target expert selected by the gating network for the input token;

[0009] The token embeddings corresponding to each token are broadcast and distributed along the ring interconnection topology, so that the target chip receives and caches the token embeddings corresponding to the local experts.

[0010] Each target kernel performs calculations on the target experts that hit the storage range of this kernel based on the cached token embedding and the expert weights stored in this kernel, and obtains the partial sum of the corresponding tokens;

[0011] Aggregate the portions corresponding to the same token along the ring interconnect topology to obtain the layer output embedding of the current MoE layer;

[0012] The load of each computing chip is monitored, and the distribution mapping of the next MoE layer is dynamically adjusted based on the load statistics.

[0013] In one implementation of the first aspect, the target experts are the Top-K target experts selected by the gating network for each input token;

[0014] Each target kernel performs calculations only on the target experts among the Top-K target experts that fall within the storage range of this kernel.

[0015] In one embodiment of the first aspect, the distribution mapping is recorded in a routing index table, which is updated by the global scheduler and broadcast to each computing core at the beginning of each MoE layer.

[0016] The entries in the routing index table include token_id, expert_id, target_die_id, weight_factor, and aggregation_target.

[0017] In one embodiment of the first aspect, broadcasting and distributing the token embeddings corresponding to each token along the ring interconnect topology includes:

[0018] The current computing chip sends the token data to the adjacent computing chip;

[0019] Upon receiving the token data, the computational core determines whether the token data corresponds to an expert stored in this core based on the distribution mapping;

[0020] When the token data corresponds to an expert stored in this core, the token data is cached in the input cache module of this core;

[0021] When the token data still needs to be transmitted to other computing chips, the token data will continue to be forwarded along the ring interconnect topology.

[0022] In one embodiment of the first aspect, the aggregation of portions corresponding to the same token and along the ring interconnect topology includes:

[0023] When the same token hits a target expert distributed across multiple computing chips, each target chip independently completes the calculation of the target expert it is responsible for, generating a partial sum of the corresponding token.

[0024] The computational core that generates a partial sum sends the partial sum generated by this core along the reverse path to the adjacent computational core.

[0025] The adjacent computing kernel will sum the received partial sum with the partial sum generated by the kernel for the same token, and continue to pass the sum along the reverse path.

[0026] After passing through multiple computational cores in the aforementioned ring interconnect topology, the layer output embedding of the same token in the current MoE layer is obtained at the aggregation endpoint core.

[0027] In one embodiment of the first aspect, the method further includes, prior to constructing the token-to-target chip distribution map:

[0028] During the hybrid expert model deployment phase, multiple experts in each MoE layer are placed on the M computing cores according to load balancing constraints.

[0029] Among them, the difference in the number of experts carried by each computing core does not exceed 1, and routing experts with high activation frequency are distributed in different computing cores based on expert popularity statistics.

[0030] The multiple experts include shared experts and routing experts; the parameters of the shared experts are stored in DRAM, and each computing chip obtains the parameters of the shared experts through a shared cache or copy-on-access method; the parameters of the routing experts are distributed and stored in different computing chips, and each computing chip stores the routing expert weights allocated to this chip.

[0031] In one embodiment of the first aspect, each target kernel performs calculations on target experts who hit the storage range of the kernel based on the cached token embedding and the expert weights stored in the kernel, including:

[0032] When the same token hits multiple target experts located on the same computing chip, the token embedding of the same token is kept resident in the input cache module of that computing chip;

[0033] The token embedding is distributed to multiple expert computing units via a local data broadcast bus, so that the multiple expert computing units perform GEMM operations on the corresponding target experts.

[0034] In one embodiment of the first aspect, monitoring the load of each computing chip and dynamically adjusting the distribution mapping of the next MoE layer based on load statistics includes:

[0035] Collect at least one of the following activation statistics: the number of tokens processed by each computing chip, the number of expert computation cycles on each computing chip, and the current idle or busy state of each computing chip.

[0036] The normalized computational load of each computational chip is determined based on the activation statistics.

[0037] When the normalized computational load of the computational core exceeds the target load threshold, the distribution mapping of at least some tokens in the next MoE layer to the target core is adjusted by updating the routing index table.

[0038] Secondly, the present invention provides a multi-core inference acceleration system for hybrid expert models, comprising M computational cores and a global scheduler, wherein the M computational cores are connected end-to-end through interconnection interfaces to form a ring interconnection topology;

[0039] The global scheduler is used to dynamically adjust the distribution mapping of the next MoE layer based on the load statistics of each computing chip.

[0040] Each computing core includes an input cache module, an expert weight storage module, a computing unit array, a routing index table, a D2D data transmission module, and a load monitoring and control module;

[0041] The input cache module is used to store the token embedding received by this core.

[0042] The expert weight storage module is used to store a subset of routing expert weights assigned to this core and local copies of shared experts;

[0043] The computing unit array is used to perform GEMM operations on target experts who hit the storage range of this core particle based on the token embedding and the expert weights, to obtain a partial sum;

[0044] The routing index table is used to store the mapping relationship between tokens, target experts, and core particles;

[0045] The D2D data transmission module is used to transmit token embedding along the forward direction of the ring interconnect topology and transmit part of it along the reverse direction of the ring interconnect topology;

[0046] The load monitoring and control module is used to collect load statistics of this core and report the load statistics to the global scheduler.

[0047] Thirdly, the present invention also provides the application of the above-described method or system in inference acceleration for hybrid expert models.

[0048] The beneficial effects of this invention include: through the input token broadcast distribution and expert cross-chip sharing mechanism under the ring interconnect architecture, token embedding can be shared at multiple points after one round of on-ring transmission among each chip, without the need for repeated reading from DRAM; through the cross-chip expert heat monitoring operator and the load state feedback mechanism based on the ring interconnect, frequently activated routing experts can be scheduled to be processed by chips with lighter loads, alleviating the "hot" problem commonly found in MoE inference. The resource idleness problem caused by "experts" is addressed; token distribution and expert output aggregation are both completed based on the ring interconnect topology, avoiding dependence on global all-to-all communication; through a pipelined overlapping design, token distribution, local expert computation, output aggregation, and next-layer distribution decisions can be performed in parallel, enabling large-scale MoE model inference to achieve pipeline scheduling at the chip / core level, significantly reducing the idle time of computation cores; through refined expert parallelism and data sharing strategies, chip area and power consumption are effectively reduced; based on the two-layer architecture of "intra-ring data sharing + inter-ring load scheduling", multiple rings can achieve collaborative load balancing through a global scheduler; when a core failure occurs in a ring, the experts on the failed ring can be redeployed to other rings through a replication mechanism, realizing the redistribution of computation tasks. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating the overall system architecture of a system according to one embodiment of the present invention.

[0051] Figure 2 This diagram illustrates the architecture of a single computing chip module in one embodiment of the present invention.

[0052] Figure 3 This diagram illustrates the internal architecture of the routing index table module in one embodiment of the present invention.

[0053] Figure 4 This diagram illustrates the internal architecture of a D2D data transmission module in one embodiment of the present invention.

[0054] Figure 5 This diagram illustrates the internal architecture of the load monitoring and control module in one embodiment of the present invention.

[0055] Figure 6 This diagram illustrates the internal architecture of the expert weight storage module in one embodiment of the present invention.

[0056] Figure 7 This is a flowchart illustrating the data processing in one embodiment of the present invention. Detailed Implementation

[0057] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0058] The Chinese meanings of some of the English terms used in this invention are as follows:

[0059] MoE: Mixture of Experts

[0060] Token: term

[0061] GEMM: General Matrix Multiply

[0062] FinDEP: Fine-grained scheduling for Disaggregated Expert Parallelism.

[0063] Token embedding: lexical embedding

[0064] TDP: Thermal Design Power

[0065] DRAM: Dynamic Random Access Memory

[0066] DMA: Direct Memory Access

[0067] Batch size: The number of samples processed at one time.

[0068] hidden dim: Hidden Dimension, the dimension of the hidden layer.

[0069] token_id: The unique identifier of the currently input token within the batch, used to identify the specific token targeted by the current routing entry, so that each core can find the corresponding distribution target based on the token ID.

[0070] expert_id: The logical number of the routing expert selected by this token in the global model, used to identify which expert needs to perform the calculation on the token.

[0071] target_die_id: The target die number to which this token needs to be distributed in this MoE layer. It is used to determine the ring broadcast target of the token embedding. Only when target_die_id matches the ID of this die will the token be cached locally and calculated.

[0072] weight_factor: The gating weight score of this token on this expert. It usually comes from the Softmax output of the MoE gating network. It is used to multiply the output vectors of each expert by this weight and then sum them up when aggregating the outputs of multiple experts to form the final output of the MoE layer.

[0073] aggregation_target: The partial and aggregation endpoint core ID of this token.

[0074] This invention provides a multi-chiplet inference acceleration method for hybrid expert models, employing a heterogeneous ring structure based on an accelerator array of M computing chiplets. Each chiplet deploys several hardware computing cores, and the chiplets are connected end-to-end via chiplet interconnect interfaces to form a ring interconnect topology. Each chiplet undertakes different computing tasks due to storing different subsets of routing experts, thus exhibiting heterogeneity in functional roles. The hardware architecture (number of computing cores, storage capacity, interconnect bandwidth, etc.) of each chiplet remains consistent to simplify design and scheduling.

[0075] This invention addresses several issues in existing MoE inference architectures. First, it employs a heterogeneous ring-based multi-core inference acceleration method. This method solves the problems of large cross-core / cross-chip all-to-all communication data volumes, severely limiting inference throughput, caused by expert parallelism. Second, it addresses the uneven load caused by significant differences in expert activation frequencies within the MoE model, where some cores are overloaded due to storing highly active experts, while others remain idle, resulting in low overall resource utilization. Third, it addresses the problem of traditional solutions where each core repeatedly loads input token data from DRAM across MoE layers, leading to off-chip memory bandwidth pressure and wasted power. Fourth, it addresses the issue of redundant on-chip memory and wasted bandwidth when multiple experts are distributed across multiple cores, as the same token embedding is repeatedly read by multiple cores due to the lack of cross-core data sharing mechanisms.

[0076] The method of this invention is based on an accelerator array containing M computing chips, which are connected end-to-end via Chiplet interconnects to form a ring interconnect topology. The method includes: for each MoE layer, creating an initial distribution map for the input token sequence based on the Top-K target experts selected by a gating network; simultaneously broadcasting each token and its embedding along the ring topology; each chip receiving a subset of input tokens from local experts, as well as tokens forwarded by neighboring chips; each chip performing GEMM operations locally on experts whose memory range matches its own, and aggregating the results along the reverse path; after completing the computation of the current MoE layer, passing the intermediate output token along the ring to the next chip, while simultaneously monitoring the load in real time; dynamically adjusting the distribution map of the next layer based on load statistics to balance the computational load of each chip; iterating the above steps until all MoE layer inference is completed. This invention also discloses a system for implementing the above method, which can be widely applied in chip computing architecture designs that support hybrid expert model inference or training.

[0077] In this invention, for each input (such as a token), only the K experts with the highest scores are activated. K determines the number of experts activated each time. Each core performs GEMM calculation only on the target experts that match this core (i.e., those experts among the Top-K experts of the current token that fall within the storage range of this core) according to the instructions of the routing index table. Experts stored in this core but not selected by any current token are not calculated.

[0078] Specifically, this invention constructs a ring-interconnected multi-chiplet computing array, introduces an input token ring-broadcasting mechanism and an expert parameter cross-chiplet sharing strategy, so that the computation of all MoE layers can be completed without frequent access to external DRAM during inference, thereby effectively reducing the frequency of off-chip memory access, alleviating the pressure on on-chip memory, and improving inference throughput. This heterogeneous ring-interconnected multi-chiplet inference acceleration method for hybrid expert models is based on an accelerator array containing M computing chipslets. Each computing chiplet deploys several hardware computing cores, and the computing chipslets are connected end-to-end through chiplet interconnect interfaces to form a ring interconnect topology; it includes the following steps:

[0079] 1. For the input token sequence, at each MoE layer, an initial distribution mapping from token to chip is constructed based on the Top-K target experts selected for each token by the gating network;

[0080] 2. Broadcast and distribute each token and its corresponding embedding to the target core along the ring interconnect topology simultaneously. Each core receives a subset of input tokens from the local expert and tokens forwarded from neighboring cores;

[0081] 3. After completing the above broadcast distribution, each kernel performs GEMM operation on the experts that hit the storage range of this kernel (i.e., those experts in the Top-K experts of the current token that fall within the storage range of this kernel), and aggregates the calculation results (partial sum) along the reverse path;

[0082] 4. After completing the calculation of the current MoE layer, the layer output embedding (i.e. the complete result of the aggregation) on each core is circulated and passed to the next core along the ring structure. At the same time, the computational load of the experts stored in the current core is monitored in real time.

[0083] 5. Based on the obtained load statistics of each core particle, dynamically adjust the distribution mapping from token to core particle in the next MoE layer to balance the computational load of each core particle in the next layer inference.

[0084] 6. Repeat steps 2 to 5 above until the inference calculations for all MoE layers are completed.

[0085] In one specific implementation, the Chiplet interconnect interface supports UCIe 3.0 or an equivalent die-to-die interconnect protocol. The UCIe 3.0 specification supports data rates of 48 GT / s and 64 GT / s, enabling high-bandwidth inter-chip communication per channel within the package, coupled with bidirectional data transmission. The ring interconnect topology between chips supports parallel data transmission in a dual-physical-ring architecture, where one ring is used for token distribution in one direction and the other for reverse aggregation of the output portion. Data flows in both directions can be parallelized within the same clock cycle, thereby increasing the effective communication bandwidth to nearly twice that of a single-ring architecture.

[0086] The dual physical rings correspond to the hardware topology level, referring to the arrangement of two physically independent UCIe links between chips / chips (e.g., a clockwise physical ring link and a counter-clockwise physical ring link). These two links are electrically isolated and each has its own independent PHY (physical layer) module, SerDes channel, and clock. The two physical links can operate independently at full bandwidth simultaneously without interference. For example, while link A (clockwise) transmits token data at 64GB / s, link B (counter-clockwise) can transmit partial and accumulated data at 64GB / s, theoretically achieving a combined bandwidth of 128GB / s.

[0087] In this invention, the aggregation endpoint of the partial sum is not fixed, but dynamically determined based on the token distribution path and load balancing strategy. Specific rules are divided into basic rules and dynamically adjusted rules.

[0088] Basic rules (without load balancing):

[0089] Assume the clockwise direction of the ring topology is "forward" (for token broadcasting and distribution), and the counter-clockwise direction is "reverse" (for partial sum aggregation). The aggregation endpoint for each token is set by default to the kernel where its initial embedding is located (i.e., the kernel where the token was first loaded). This is because after the calculations in the MoE layer are completed, the aggregation result needs to be returned to the token's "owner" kernel to facilitate processing in the next layer. The aggregation process is as follows: all kernels storing the selected expert for this token calculate their partial sums, then accumulate them level by level in a counter-clockwise direction, finally passing the result to the initial kernel. This initial kernel outputs the complete layer output embedding.

[0090] Dynamically adjust rules (when load balancing is enabled):

[0091] When the global scheduler decides to assign a portion of a thermal expert's computational task to a replica of another core, the partial sum and aggregation endpoint of that token may need to be adjusted accordingly. For example, the initial core of token T is C0, but one of the thermal experts' computations is scheduled to be performed on a replica of core C5. In this case, the aggregation endpoint might be reset to C5 (to reduce the reverse transmission distance), or the final accumulation at C0 might need to wait for the partial sum from C5 to be transmitted via a ring. The specific decision is made by the scheduler based on the principle of minimizing communication overhead. The scheduler adds an `aggregation_target` field to the routing index table for each token, recording the aggregation endpoint core ID field. Each core uses this field to determine which core to pass the accumulated partial sum to.

[0092] The aggregation endpoints for different tokens can be different, depending on the initial kernel and load scheduling decisions for each token.

[0093] The design of specifying an independent destination for each token is achieved through the aggregation_target field in the routing index table. Each kernel determines the next hop direction based on this field during forwarding.

[0094] In this invention, partial sums represent the accumulated intermediate data before aggregation; layer output embeddings represent the output data after aggregation. During the aggregation process, multiple partial sums are accumulated step by step to ultimately form an intermediate output token.

[0095] Furthermore, during the MoE model deployment phase, all experts are partitioned across M computing cores according to load balancing constraints. Specifically, for a MoE layer containing S experts (including 1 shared expert and S-1 routing experts), integer linear programming is used to distribute the S experts across the M cores, ensuring that the difference in the number of experts carried by each core does not exceed 1. Furthermore, routing experts with high activation frequencies are distributed across different cores based on expert popularity statistics to avoid a single core becoming a hot bottleneck. Taking DeepSeek-V3 as an example, if S=257 (256 routing experts + 1 shared expert), and M=16, then each core is allocated 16 routing experts, and each core also contains a local copy of the shared expert. The parameters of the shared expert are stored only once in DRAM, and each core obtains them through a shared cache or copy-on-access method; the routing expert parameters are stored completely distributed, with each core storing only the weights of the 16 routing experts it is responsible for. The term "high activation frequency" refers to the number of times a particular expert is activated within a given time window (such as the most recent N MoE layers or the most recent T token batches), exceeding a set multiple of the average activation frequency of all experts. In this invention, "high activation frequency" refers to the number of times a routing expert is activated, within a set time window, exceeding 1.8 times the average activation frequency of all routing experts. This multiple can be configured via a register, with a typical value range of 1.5 to 2.5.

[0096] Each core component's load monitoring module contains an expert activation frequency counter. Each time a token is used to invoke an expert, the corresponding expert's counter is incremented by 1. At the end of each MoE layer, each core component reports its counter value to the global scheduler. The scheduler aggregates the activation frequencies of all experts, calculates the global average frequency freq_avg, and then, based on a preset high threshold coefficient α (e.g., 1.8), determines which experts satisfy freq_expert > α × freq_avg, marking them as high-activation-frequency experts.

[0097] During broadcast distribution, each chiplet caches a subset of input tokens assigned to its experts, and simultaneously forwards received token data from other chiplets to adjacent chiplets along the ring topology. Specifically, the ring broadcast distribution uses a "pipelined" model: in the first clock cycle, chiplet 0 sends its local token data to chiplet 1; in the next clock cycle, chiplet 0 sends the next batch of tokens, while chiplet 1 forwards the recently received tokens to chiplet 2, and so on. After the input token data completes M-1 transmissions in the ring structure, all chiplets receive all the input tokens required for processing by their respective experts, eliminating the need to repeatedly read input data from external DRAM. In this way, the token distribution-computation-aggregation-redistribution process during MoE inference can be performed in a fully pipelined manner between chiplets, significantly reducing the frequency of access to off-chip DRAM token data. The caching in this invention primarily caches the token embedding.

[0098] During the GEMM computation process after distribution, each core performs batch scheduling of computation tasks for multiple experts stored on the same core when executing local expert GEMM operations. Specifically, each core contains multiple vector processing units or tensor kernels, capable of processing up to P independent matrix multiplications simultaneously. For the same token potentially hitting multiple experts located in the same core (e.g., in DeepSeek-V3, each token selects 8 routing experts, of which 2-3 may fall into the same core), this invention supports keeping the token's embedding resident in the core's input cache and simultaneously distributing it to multiple expert computation units via the local data broadcast bus, achieving local data sharing among multiple experts and reducing redundant multiplication operations. Specifically, if E experts on a core need to process the same token, the traditional method requires reading the token embedding from the cache E times, while this invention only requires reading it once, thus saving (E-1) / E input cache read bandwidth.

[0099] Furthermore, the same token may hit experts distributed across multiple cores (this is normal in MoE, as each core stores a different subset of experts). In this case, each core independently completes the calculation for its assigned expert, generating a partial output for the corresponding token. These partial outputs need to be accumulated to form the final output of the token in this MoE layer. This invention adopts an "aggregation along a reverse loop path" strategy: after the calculation is completed, core 0 sends its partial sum to core M-1, core M-1 adds its own partial sum and sends it to core M-2, and so on, eventually converging to the designated endpoint. This aggregation process can partially overlap with the token distribution of the next layer.

[0100] In this invention, the definitions of forward and reverse are as follows:

[0101] Forward Direction: Clockwise direction, i.e., the direction of increasing core number (0→1→2→…→M-1→0). This direction is used for broadcast distribution of token embeddings. Data is sent from the current core to the core with number +1 (modulo M).

[0102] Backward Direction: Counter-clockwise direction, i.e., the direction of decreasing core number (0→M-1→M-2→…→1→0). This direction is used for the aggregation and accumulation of partial sums. Data is sent from the current core to the core with number -1 (modulo M).

[0103] In this invention, the dynamic adjustment of the distribution mapping and load balancing is implemented by a global scheduler. The global scheduler can be integrated into a master control chip or communicate with each chip as an independent control module via a sideband channel. The scheduler dynamically adjusts the distribution mapping strategy of the next layer between MoE layers by monitoring the activation statistics of experts on each chip (including but not limited to: the number of tokens processed by each chip, the number of computation cycles for each expert, and the current idle / busy state of each chip). Specifically, the scheduler maintains a chip load vector L = [l_0, l_1,..., l_{M-1}], where l_i (i ranges from 0 to M-1) represents the normalized computational load of chip i in the current MoE layer (actual computation cycles used / total available cycles). For the next MoE layer, the scheduler calculates the ideal load target L_target = (total number of tokens). Number of experts activated per token / (M) The computational capacity of each core. For loads exceeding a certain target threshold (e.g., greater than 1.2). For a given kernel (L_target), the scheduler allocates some of the input tokens of high-traffic experts stored on that kernel to other kernels with lower loads. There are two allocation methods: one is to temporarily store copies of the weights of these high-traffic experts in the kernel's local weight copy cache (suitable for short-term load fluctuations); the other is to immediately transfer the tokens to the kernel storing the expert's weights via a ring interconnect for remote calculation (suitable for long-term load offsets). Both methods are implemented by updating the routing index table.

[0104] In a specific implementation, the ring interconnect transmission in this invention will not be transmitted to the original expert, and experts with excessive load need to be excluded.

[0105] When the global scheduler detects that a specific expert stored on a core is overloaded due to excessive activation frequency, the scheduler first marks the expert as an overheated expert and excludes it from subsequent allocations for that core (i.e., no more tokens requiring processing by that expert will be allocated to that core). Subsequently, the scheduler reassigns these tokens to other cores storing copies of that expert, and the token embeddings are transmitted to the target cores via a ring interconnect for remote computation. The expert weights on the original core remain resident during this period and can be reactivated once the load recovers, achieving elastic scheduling.

[0106] In one specific embodiment of the present invention, when a computing kernel becomes overloaded due to excessively high activation frequency of a particular expert, the global scheduler can distribute the next batch of tokens to be processed by the overheated expert to other kernels storing copies of that expert, or migrate some low-activation experts to kernels with lighter loads. The maintenance of expert copies is achieved through the primary weight stored in DRAM and the backup weight cached on the target kernel. The backup weight is pre-written during model loading or copied from the primary kernel on demand via a UCIe link during inference. The additional communication overhead of the copying operation is acceptable relative to the throughput gains from load balancing. In this invention, a low-activation expert is defined as an expert whose activation count is less than 0.4 times the average activation count of all routing experts. This ratio can be configured via registers, with a typical value range of 0.3 to 0.6. The low-activation threshold coefficient β has a default value of 0.4 and a range of 0.3 to 0.6. If freq_expert < β × freq_avg, it is determined to be a cold expert (low-activation expert) and is given priority as a migration candidate. The global scheduler can prioritize migrating the weights of these low-activity experts to lighter-loaded granules to free up storage resources in the original granules and optimize load distribution.

[0107] The threshold coefficients α / β for determining high / low activation experts can be dynamically adjusted via the control bus to adapt to different inference load modes.

[0108] In this invention, the migration includes a complete set of parameters (expert weights) and metadata associated with the expert (routing index, activation statistics, etc.). However, the completeness of the migration content varies depending on the type of migration, specifically falling into two categories:

[0109] 1. Expert replica creation (short-term load balancing) is suitable for scenarios with small load fluctuations and no need for permanent relocation. Migration includes:

[0110] Expert weight parameters: The weight data of the low-activation expert is copied from the original core to the target core. In one implementation, each logical expert can correspond to multiple physical copies. For example, according to the EPLB architecture of vLLM, each logical expert can have multiple physical copies distributed on different cores. These copies are pre-configured or created online during the model loading phase.

[0111] Routing Index Table Entry: Add a new record to the routing index table of the target kernel, associating the expert ID with the target kernel ID. Meanwhile, the corresponding routing entry in the original kernel can be retained or marked as "inactive".

[0112] Activate statistics: Synchronize the expert's historical activation frequency and other statistical data to the target chip's load monitoring module as a reference for subsequent load balancing decisions.

[0113] Expert status flag: Marks the expert as "active" on the target chip.

[0114] 2. Complete expert migration (long-term load balancing / fault tolerance)

[0115] This is suitable for scenarios where persistent load offsets require permanent adjustments. In addition to everything in Migration 1, it also includes:

[0116] In-place deletion / invalidation mark: Mark the routing entry for this expert in the original core as "invalid", and subsequent tokens will no longer be routed to the original location.

[0117] Original nucleus weight cache release or reuse: The expert weight cache on the original nucleus can be released or reused for other experts to reclaim storage resources.

[0118] Global consistency update: Ensure that the mapping relationship of the expert is consistent in the routing index table of all cores.

[0119] In some specific embodiments, the present invention also provides a system for implementing the above method, comprising M computational cores (M being a positive integer power of 2, typically 8, 16, 32, or 64), each computational core containing the following modules:

[0120] (1) Input Buffer: This module stores the token embedding data received by this core. The buffer is organized into a multi-bank structure, supporting parallel reading by multiple expert computing units. The buffer size is designed to accommodate at least all token embeddings in a batch that need to be processed by the experts of this core. The specific capacity depends on the batch size and hidden_dim.

[0121] (2) Expert Weight Storage Module: This module stores a subset of routing expert weights assigned to the core and local copies of shared experts. It is typically implemented using high-speed SRAM and supports pipelined provision of weight data during GEMM computation. For a typical DeepSeek-V3 configuration (16 routing experts per core, each expert weight dimension being hidden_dim×dff, fp16 format), the expert weight storage capacity is approximately 16×4096×14336×2 bytes ≈ 1.8GB. In actual chip designs, it is usually distributed across multiple banks.

[0122] (3) Compute Array: Contains multiple matrix multiplication accelerators (such as systolic arrays or tensor kernels) for performing GEMM operations between local experts and input tokens. Each compute unit can work independently or collaboratively process larger matrix dimensions of the same expert. In this invention, multiple compute units can process different experts simultaneously, achieving expert-level parallelism; when multiple experts on the same core need to process the same token, the compute array can reuse the embedding data of that token.

[0123] (4) Routing Index Table: This table stores the mapping relationship between tokens, experts, and die particles. It is updated by the global scheduler and broadcast to all die particles at the beginning of each MoE layer. Each entry in the table contains: [token_id, expert_id, target_die_id, weight_factor, aggregation_target]. Each die particle uses this table to determine whether a received token belongs to an expert it is responsible for, and whether it needs to forward some tokens to expert copies of other die particles.

[0124] (5) D2D Data Transmission Module (Die-to-Die Transport): Implements UCIe 3.0 or equivalent physical and link layer protocols, supporting bidirectional data flow over a ring interconnect. This module includes a transmit FIFO, a receive FIFO, a link training state machine, a packet encapsulation / decapsulation unit, and arbitration logic. To support broadcast distribution and reverse aggregation in this invention, the D2D module is equipped with two independent transmission engines: one for pushing token data (packet type: token_data) clockwise and the other for pushing partial sums (packet type: partial_sum) counterclockwise. The two engines share the same physical link but use different virtual channels, conforming to the flow control mechanism of the UCIe standard.

[0125] The virtual channel, corresponding to the protocol / link layer, refers to multiple independent communication channels logically divided on the same physical link through time-division multiplexing or packet marking. Each virtual channel has its own independent flow control (FC) mechanism, buffer queue (FIFO), and quality of service (QoS) priority. Multiple virtual channels share the same physical bandwidth, but are allocated in a time-sharing manner through a scheduling algorithm. For example, VC0 (virtual channel 0) is used for token broadcasting with high priority, while VC1 is used for partial and aggregated data with low priority. Data is sent first when VC0 (virtual channel 0) has data, and can only be sent when VC1 (virtual channel 1) is idle. The UCIe standard itself defines the virtual channel mechanism, allowing multiple data streams to be multiplexed on the same physical link.

[0126] (6) Load Monitoring and Control Module: This module collects real-time computing load statistics for this core, including but not limited to: computing unit utilization, unprocessed token queue depth, expert activation frequency, etc. This module reports the statistical data to the global scheduler through the sideband network (such as a low-speed data bus or control bus) and receives configuration updates from the scheduler (such as routing table updates, expert replica enabling, etc.).

[0127] The present invention also provides a chip that includes a multi-core inference acceleration system for hybrid expert models.

[0128] Figure 1 This is a schematic diagram of the system architecture in one embodiment of the present invention. The diagram shows four computing chips (chiplet_0 to chiplet_3) connected end-to-end via a UCIe interface to form a ring interconnect topology. Each chiplet contains an input buffer, expert weight storage, a computing unit array, a routing index table, a D2D data transmission module, and a load monitoring and control module. The global scheduler can be located inside chiplet_0 or connected as a separate chip via a dedicated interface. Figure 2 The diagram shows the composition of a single computing core, detailing its internal structure and data flow between modules: the input buffer is connected to the computing unit array via an internal bus; the expert weight storage module directly provides weight data to the computing unit array; the D2D module receives data packets from the previous core, parses them, and caches the token in the local input buffer or forwards it directly according to the routing index table; after calculation, the partial sum is sent to the previous core through the reverse channel of the D2D module; the load monitoring module reads the status from the computing unit array and the input buffer and outputs it externally through the control bus.

[0129] Figure 2(Diagram showing the internal module composition and interconnections of a single computing die) This diagram details the internal module composition of a computing die, including: a D2D data transmission module (containing Tx / Rx FIFO and virtual channels), an input buffer module (multi-bank SRAM), a computing unit array (16 matrix computing cores), a routing index table (SRAM), an expert weight storage module (multi-bank SRAM, containing routing expert weights and shared expert copies), and a load monitoring and control module. Solid arrows represent data flow (including token embedding, weight data, and partial sums), while dashed arrows represent control flow (routing table updates and load statistics reporting).

[0130] Figure 3 The diagram (Internal structure of the routing index table module) illustrates the hardware implementation details of the routing index table. It includes: a configuration register interface (base address register, routing table depth register, update enable register), an SRAM controller, a dual-port routing table SRAM (each entry contains five fields: token_id, expert_id, target_die_id, weight_factor, and aggregation_target), and output parsing and hit / miss logic (comparing target_die_id with the local die ID to generate a hit / miss flag). This module receives routing table updates from the global scheduler via the control bus and provides fast lookup when a token arrives.

[0131] Figure 4 (D2D Data Transmission Module Internal Structure Diagram) illustrates the internal structure of a D2D module based on UCIe 3.0. The diagram is divided into receive and transmit paths: the receive path includes deframing / CRC checking, a virtual channel demultiplexer, Rx VC0FIFO (token data) and Rx VC1 FIFO (partial sum), arbitration and routing distribution logic; the transmit path includes Tx VC0FIFO (token broadcast), Tx VC1 FIFO (partial sum), a virtual channel multiplexer, and framing / CRC generation. The D2D data transmission module connects to the UCIe PHY. This design supports independent transmission on two physical rings (clockwise and counterclockwise), with data types distinguished within each physical ring via virtual channels.

[0132] Figure 5 The diagram illustrates the internal components of the load monitoring and control module. These include: a statistical acquisition interface (containing utilization counters for 16 compute cores, an expert activation histogram generator, input buffer queue depth monitoring, and a temperature / voltage sensor readout interface), statistical summarization and formatting logic, and interrupt trigger configuration logic. This module collects load information from each core in real time, calculates normalized load metrics, and reports them to the global scheduler via the control bus.

[0133] Figure 6 The diagram illustrates the internal structure of the expert weight storage module. It includes: weight loading control logic (supporting loading routing expert weights from DRAM via DMA, receiving expert replicas from adjacent chips via UCIe, and dynamic hot expert replica refresh), a weight storage array (multi-bank SRAM, each bank corresponding to one routing expert weight), a weight distribution network (Crossbar), and a shared expert replica area (independent of the routing expert bank). This module is connected to the computing array via a weight data bus, supporting parallel reading of multiple expert weights.

[0134] Figure 7 The flowchart of the method of this invention is as follows, and the main steps include: Start → Model deployment and expert partitioning (offline) → Load input token sequence → Initialize routing table → Enter MoE layer loop (for each layer): ① Gated network calculates the Top-K experts for each token → ② Construct initial distribution mapping → ③ Start ring broadcast distribution (iteration M times) → ④ Each core executes local expert GEMM (parallel) → ⑤ Reverse ring aggregation of partial sums → ⑥ Update output tokens → ⑦ Collect load statistics → ⑧ Dynamically adjust the next layer distribution mapping → Determine if it is the last layer: if not return ③ If so, then the reasoning ends.

[0135] Compared with existing related technologies, the advantages of the present invention may include the following:

[0136] (1) Significantly reduces off-chip memory access and on-chip memory overhead. Through input token broadcasting and distribution under the ring interconnect architecture and expert cross-chip sharing mechanism, token embedding can be shared at multiple points after one round of on-ring transmission among each chip, without the need for repeated reading from DRAM. In existing MoE inference schemes, the amount of token data that a single chip needs to load in each MoE layer is approximately B. H F (K / M) bytes (B is the batch size, H is the hidden layer dimension, F is the number of experts processed per chip, and K / M is the selected expert ratio), this invention reduces this repetitive loading to approximately B bytes. Loading H bytes at once can reduce off-chip storage access by a factor of (L×K / M) in scenarios with a large number of MoE layers, where L is the total number of MoE layers.

[0137] (2) Effectively balance the computational load between core particles. Through the cross-core expert hotness monitoring operator and the load status feedback mechanism based on ring interconnection, the routing experts that are activated at high frequency can be scheduled to be processed by core particles with lighter loads, which alleviates the resource idle problem caused by "hot experts" that is common in MoE inference. The overall core particle utilization is expected to be improved by 15%~30% (based on the quantitative analysis of existing MoE acceleration schemes).

[0138] (3) Reduce all-to-all communication overhead between chiplets. In this invention, token distribution and expert output aggregation are both completed based on the ring interconnect topology, avoiding dependence on global all-to-all communication. Research on the MoEntwine scheme shows that the cross-chiplet communication volume can be reduced by up to 62% based on the ring mapping design. On this basis, this invention further integrates the ring topology with the data sharing mechanism, and the communication overhead is expected to be reduced by more than 15%.

[0139] (4) Supports fully pipelined MoE inference. Through a pipelined overlapping design, token distribution, local expert computation, output aggregation, and next-layer distribution decisions can be performed in parallel, enabling pipelined scheduling of large-scale MoE model inference at the chip / core level, significantly reducing the idle time of computation cores. Compared with traditional GPU solutions, the expected inference throughput can be increased by 1.5 to 2 times.

[0140] (5) On-chip storage resource utilization efficiency is significantly improved. Expert Streaming research shows that on chiplet-based multi-chip architecture, a 78.8% on-chip storage saving can be achieved through refined expert parallelism and data sharing strategies, effectively reducing chip area and power consumption.

[0141] (6) Good scalability and fault tolerance. Based on the two-layer architecture of "intra-ring data sharing + inter-ring load scheduling", a single ring structure can contain multiple (e.g., 4 / 8 / 16) computing cores. Multiple rings can achieve collaborative load balancing through a global scheduler. When a core of a ring fails, the experts on the failed ring can be redeployed to other rings through a replication mechanism to achieve the redistribution of computing tasks.

[0142] Example 1: Cloud-based online inference service for large language models (based on DeepSeek-V3)

[0143] 1. Purpose and Background of the Scene

[0144] A cloud service provider plans to deploy an online inference service for its DeepSeek-V3 model (671B total parameters, 37B activation parameters) on its AI inference platform for general natural language processing tasks (such as document summarization, code generation, multi-turn dialogue) for enterprise users. This service requires support for high-concurrency requests (peak 1000+ queries per second) with end-to-end inference latency of no more than 200 milliseconds per request (to provide a smooth user experience). Traditional GPU cluster solutions (such as 8×H100), while offering strong performance, are expensive (hundreds of thousands of dollars per node), power-hungry, and have insufficient resource utilization (due to cross-card communication bottlenecks and uneven load distribution caused by MoE expert parallelism), failing to meet the service provider's engineering requirements for high throughput, low latency, and low cost per node.

[0145] This embodiment aims to verify whether the 16-core ring interconnect MoE acceleration chip proposed in this invention can achieve or exceed the above performance indicators under typical cloud inference loads, and demonstrate advantages over traditional GPU solutions in terms of cost, power consumption and resource utilization.

[0146] 2. Test Environment and Detailed Configuration

[0147]

[0148] 3. Detailed Test Results and Data Analysis

[0149] Throughput and latency:

[0150] The chip of this invention, under stable load (800 req / s), has an average dynamic batch size of 16 and an average end-to-end latency (from request enqueuing to complete output generation) of 165 milliseconds (198 milliseconds for P95 and 215 milliseconds for P99). Single-node throughput reaches 812 req / s, corresponding to a token throughput of approximately 156K tokens / s (an average of 192 output tokens per request). In the 8×H100 baseline, under the same load, latency is significantly increased (average 298 milliseconds, P95 400+ milliseconds), throughput is only about 450 req / s (limited by cross-card all-to-all communication bottlenecks), and token throughput is approximately 86K tokens / s.

[0151] Resource utilization rate:

[0152] The average utilization rate of each chip in this invention is 82%, and the difference between the highest and lowest utilization chip (load imbalance) is an average of 12% (thanks to the dynamic load balancing mechanism). 8×H100: The utilization rate of each GPU fluctuates greatly (40%~95%), with an average imbalance of 38%, and some GPUs are idle while waiting for cross-card communication.

[0153] Energy efficiency and cost:

[0154] The chip of this invention has a total power consumption of 385W (including HBM), with a performance-to-power ratio of approximately 2.11 req / s / W. The 8×H100 system has a power consumption of approximately 2800W (including GPU and system cooling), with a performance-to-power ratio of approximately 0.16 req / s / W, about 1 / 13 of that of the chip of this invention. Hardware procurement costs: The estimated cost per chip for the prototype of this invention is approximately US$3,500 (in mass production); the cost of an 8×H100 node is approximately US$250,000. The throughput per dollar of this invention is approximately 18 times that of the baseline.

[0155] 4. Conclusion

[0156] In cloud-based DeepSeek-V3 inference scenarios, this chip achieves 1.8 times the throughput of the baseline solution at a significantly lower cost and power consumption than traditional GPU clusters, while keeping P99 latency below 215 milliseconds, meeting the stringent latency requirements of online services. The dynamic load balancing mechanism significantly reduces inter-chip load unevenness caused by expert hotspots (unevenness reduced from 38% to 12%), while the ring broadcast distribution mechanism eliminates cross-card all-to-all communication bottlenecks, enabling a single chip to efficiently support real-time inference of 671B MoE models. This solution provides cloud service providers with a highly competitive, low-cost, low-latency, high-throughput alternative for large model inference.

[0157] Example 2: Real-time voice assistant at the edge (based on Qwen2-57B-A14B)

[0158] 1. Purpose and Background of the Scene

[0159] The next-generation in-vehicle voice assistant integrates a large language model for natural voice interaction (command understanding, navigation search, vehicle status control, etc.). The in-vehicle environment has extremely high requirements for low latency and low power consumption: the interaction latency must be less than 100 milliseconds to avoid perceived lag; simultaneously, due to limited in-vehicle power supply and heat dissipation capabilities, the peak power consumption of the inference chip must not exceed 50W. Furthermore, the model parameter size needs to be controlled to accommodate embedded storage capacity (≤32GB). The Qwen2-57B-A14B MoE model (64 routing experts, each token activating 4 experts) is considered one of the candidate models for the in-vehicle field due to its relatively small activation parameters (approximately 14B) and moderate parameter count (57B). However, efficiently running this model on a single low-power chip still faces challenges: expert parallelism is difficult to achieve on low-power devices (multi-chip communication overhead is too high), and traditional methods require repeatedly loading a large amount of input data, placing a significant burden on off-chip DRAM bandwidth and power consumption.

[0160] This embodiment uses a simplified version of the architecture of the present invention (4-core ring interconnect). Under the power consumption limit of 50W, it demonstrates whether real-time inference of the Qwen2-57B model can be achieved and the end-to-end latency target of ≤100ms can be met, illustrating the application process of the method of the present invention in low-power multi-core systems.

[0161] 2. Test Environment and Detailed Configuration

[0162]

[0163] 3. Detailed Test Results and Data Analysis

[0164] Latency test (single request, 15 input tokens, 30 output tokens). The chip of this invention has an end-to-end average latency of 78 milliseconds. Breakdown is as follows: input embedding loading and initial distribution ~5ms, 28 MoE layers (each with a latency of ~2.2ms) totaling ~61.6ms, and the remaining decoding stage (serial generation of output tokens) ~11.4ms. The most critical first token time (Time To First Token, TTFT) is 29 milliseconds.

[0165] Jetson Orin baseline: Due to the lack of expert parallel support on a single chip, the activation parts of all experts must be calculated sequentially, resulting in extremely slow inference: end-to-end latency of 2.8 seconds and TTFT of about 1.2 seconds, which cannot meet the requirements of real-time interaction.

[0166] Power consumption and thermal performance:

[0167] When running the aforementioned workload, the chip of this invention consumes an average of 38W and a peak of 45W, with the chip junction temperature remaining stable at 82℃ (under passive cooling conditions), which does not exceed the safety threshold. In contrast, the Jetson Orin chip, running the same model, consumes 43W but reaches a junction temperature as high as 96℃ (with active fan cooling) due to frequent DRAM accesses and low computational efficiency, and its throughput is insufficient.

[0168] Energy efficiency comparison:

[0169] The chip of this invention has an energy efficiency of 0.337 requests / joule in this scenario (with a 78ms latency, an average power consumption of 38W, approximately 2.96 joules of energy per request, and processing approximately 1000 / 78≈12.82 requests per second, the energy efficiency is 12.82req / s / 38W = 0.337 requests / joule, meaning that 0.337 requests can be processed per joule of energy, or a throughput of approximately 0.337 requests / second / watt per watt). The Jetson Orin can only process 0.357 requests per second, with a power consumption of 43W, and an energy efficiency of 0.357 / 43 = 0.0083 requests / joule, meaning a throughput of only 0.0083 per watt. The energy efficiency of this invention is 0.337 / 0.0083 = 40.6 times that of the baseline. In terms of throughput per watt, this invention is 0.337 req / (s·W), while the baseline is 0.0083. The difference in req / (s·W) is also 40.6 times.

[0170] 4. Conclusion

[0171] This invention, using a simplified 4-chip architecture, successfully achieved automotive-grade real-time inference of the Qwen2-57B-A14B MoE model under a 45W power consumption limit, with a latency as low as 78 milliseconds, fully meeting the interaction response speed requirements of voice assistants. Compared to embedded GPU solutions (Jetson Orin), this invention utilizes ring broadcast distribution to share input token data among chips, significantly reducing the number of off-chip DRAM accesses, thereby achieving two orders of magnitude latency optimization and a 40-fold energy efficiency improvement at the same power consumption. This embodiment fully demonstrates the enormous potential of this invention's architecture in low-power real-time inference scenarios at the edge.

[0172] Example 3: Multi-node collaborative large model training in a data center (based on DeepSeek-R1)

[0173] 1. Purpose and Background of the Scene

[0174] Large-scale instruction fine-tuning (SFT) and reinforcement learning training are performed using the DeepSeek-R1 model. This model employs a MoE architecture similar to DeepSeek-V3, but due to the addition of the Thought Chain (CoT) inference mechanism, it requires handling longer contexts (sequence lengths can reach 32K) during training. The training task demands extremely high computational throughput and cross-node communication bandwidth, typically requiring a cluster of hundreds of GPUs. However, the traditional data parallelism + expert parallelism approach has two problems: first, the amount of cross-node all-to-all communication data grows linearly with the number of experts, becoming a scalability bottleneck; second, due to the dynamic changes in expert load, GPU utilization typically fluctuates between 40% and 70%, wasting significant resources.

[0175] This embodiment attempts to use chips employing the architecture of this invention to form a 32-node cluster (each node has one 16-chip), and constructs a hierarchical ring structure through chip-level inter-ring interconnection technology to verify whether this scheme can improve communication efficiency and resource utilization in distributed training of the MoE model.

[0176] 2. Test Environment and Detailed Configuration

[0177]

[0178] 3. Detailed Test Results and Data Analysis

[0179] Training swallowing:

[0180] The 32-node cluster of this invention achieves a training throughput of 2.4 million tokens per second under stable conditions. The intra-node computation efficiency is 78% of the peak, the average inter-node communication latency is 18 microseconds, and the amount of cross-node communication data is reduced by approximately 55% compared to the baseline (thanks to ring broadcast distribution and load balancing).

[0181] 128×H100 baseline: Total throughput 1.8M tokens / second (H100 single card has lower effective throughput in MoE training). Although H100 single card has higher computing power, its total throughput is actually lower than that of the 32-node version of this invention because cross-node all-to-all communication overhead accounts for 23% of the total training time (compared to only 9% in this invention), and uneven expert load leads to a utilization rate of 65%.

[0182] Quantitative analysis of communication overhead:

[0183] During the forward and backward training of a MoE layer, the baseline scheme requires approximately the following amount of token-expert allocation information to be transmitted: (batch_seq_len × number of activated experts) × hidden_dim × 2. For a 32K sequence, with a global batch of 512 (i.e., 16M tokens), each token activates 8 experts, requiring 128M routing records. Each record carries embedding (4096 × 2B = 8KB), resulting in approximately 1TB of raw data. After all-to-all transmission, the total cross-card traffic is approximately 8TB (because each expert needs to aggregate all tokens). This invention employs ring broadcast + local replication. After each token's data is broadcast once within the ring, all kernels can access it. The total cross-node traffic is only 1.2 times the original data volume (considering some redundancy), approximately 1.2TB, a reduction of 85% compared to the baseline.

[0184] Actual measurements: The baseline takes 460ms to communicate per iteration cycle (approximately 2 seconds), while the present invention takes 180ms.

[0185] Resource utilization rate:

[0186] The average utilization rate of the chip in this invention is 89% (the load balancing algorithm makes the imbalance less than 8%). The average utilization rate of the H100 GPU is 67% (some GPUs are overloaded due to processing thermal experts, while the rest are idle and waiting).

[0187] Cost and energy efficiency:

[0188] The total power consumption of this invention with 32 nodes is approximately 12.8 kW (32 × 400 W), with a performance-to-power ratio of 187.5 tokens / J. The total power consumption of the 128 × H100 system is approximately 128 × 350 W = 44.8 kW, with a performance-to-power ratio of 40.2 tokens / J. The energy efficiency of this invention is 4.66 times that of the baseline. Hardware cost: The estimated cost of this invention with 32 nodes is approximately US$1.12 million (32 × US$35,000), and that of the 128 × H100 system is approximately US$4 million. This invention achieves a cost-effectiveness ratio of 3.57 times.

[0189] 4. Conclusion

[0190] In large-scale MoE model distributed training scenarios, the 32-node cluster of this invention achieves 33% higher training throughput (2.4M vs 1.8M tokens / s) than the 128×H100 cluster, with a quarter of the hardware cost and about one-third of the power consumption. The core advantages are: the ring architecture and data sharing mechanism significantly reduce cross-node communication (by 85%), and dynamic load balancing increases resource utilization from 67% to 89%. This embodiment demonstrates that this invention is not only applicable to inference but can also be extended to a highly efficient multi-node training system, significantly reducing the threshold and cost of training large models.

[0191] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0192] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Clearly, those skilled in the art can make various alterations and variations to this application without departing from its spirit and scope. Thus, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0196] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A multi-core grain reasoning acceleration method for hybrid expert models, characterized in that, The method is based on an accelerator array containing M computing chips, which are connected end-to-end via interconnect interfaces to form a ring interconnect topology, including: For the input token sequence, in the current MoE layer, a distribution mapping from token to target kernel is constructed based on the target expert selected by the gating network for the input token; The token embeddings corresponding to each token are broadcast and distributed along the ring interconnection topology, so that the target chip receives and caches the token embeddings corresponding to the local experts. Each target kernel performs calculations on the target experts that hit the storage range of this kernel based on the cached token embedding and the expert weights stored in this kernel, and obtains the partial sum of the corresponding tokens; Aggregate the portions corresponding to the same token along the ring interconnect topology to obtain the layer output embedding of the current MoE layer; The load of each computing chip is monitored, and the distribution mapping of the next MoE layer is dynamically adjusted based on the load statistics.

2. The method of claim 1, wherein, The target experts are the Top-K target experts selected by the gating network for each input token; Each target kernel performs calculations only on the target experts among the Top-K target experts that fall within the storage range of this kernel.

3. The method as described in claim 1, characterized in that, The distribution mapping is recorded in the routing index table, which is updated by the global scheduler and broadcast to each computing chip at the beginning of each MoE layer. The entries in the routing index table include token_id, expert_id, target_die_id, weight_factor, and aggregation_target.

4. The method as described in claim 1, characterized in that, The step of broadcasting and distributing the token embeddings corresponding to each token along the ring interconnection topology includes: The current computing chip sends the token data to the adjacent computing chip; Upon receiving the token data, the computational core determines whether the token data corresponds to an expert stored in this core based on the distribution mapping; When the token data corresponds to an expert stored in this core, the token data is cached in the input cache module of this core; When the token data still needs to be transmitted to other computing chips, the token data will continue to be forwarded along the ring interconnect topology.

5. The method as described in claim 1, characterized in that, The aggregation of the portion corresponding to the same token and along the ring interconnect topology includes: When the same token hits a target expert distributed across multiple computing chips, each target chip independently completes the calculation of the target expert it is responsible for, generating a partial sum of the corresponding token. The computational core that generates a partial sum sends the partial sum generated by this core along the reverse path to the adjacent computational core. The adjacent computing kernel will sum the received partial sum with the partial sum generated by the kernel for the same token, and continue to pass the sum along the reverse path. After passing through multiple computational cores in the aforementioned ring interconnect topology, the layer output embedding of the same token in the current MoE layer is obtained at the aggregation endpoint core.

6. The method as described in claim 1, characterized in that, Before constructing the distribution mapping from token to target kernel, the method further includes: During the hybrid expert model deployment phase, multiple experts in each MoE layer are placed on the M computing cores according to load balancing constraints. Among them, the difference in the number of experts carried by each computing core does not exceed 1, and the routing experts are distributed in different computing cores based on the expert popularity statistics. The multiple experts include shared experts and routing experts; the parameters of the shared experts are stored in DRAM, and each computing chip obtains the parameters of the shared experts through a shared cache or copy-on-access method; the parameters of the routing experts are distributed and stored in different computing chips, and each computing chip stores the routing expert weights allocated to this chip.

7. The method as described in claim 1, characterized in that, Each target kernel performs calculations on target experts that hit the storage range of its target kernel, based on the cached token embedding and the expert weights stored in this kernel. This includes: When the same token hits multiple target experts located on the same computing chip, the tokenembedding of the same token is kept resident in the input cache module of that computing chip; The token embedding is distributed to multiple expert computing units via a local data broadcast bus, so that the multiple expert computing units perform GEMM operations on the corresponding target experts.

8. The method as described in claim 1, characterized in that, The monitoring of the load on each computing chip and the dynamic adjustment of the distribution mapping for the next MoE layer based on load statistics include: Collect at least one of the following activation statistics: the number of tokens processed by each computing chip, the number of expert computation cycles on each computing chip, and the current idle or busy state of each computing chip. The normalized computational load of each computational chip is determined based on the activation statistics. When the normalized computational load of the computational core exceeds the target load threshold, the distribution mapping of at least some tokens in the next MoE layer to the target core is adjusted by updating the routing index table.

9. A multi-core inference acceleration system for hybrid expert models, characterized in that, It includes M computing cores and a global scheduler. The M computing cores are connected end to end through interconnection interfaces to form a ring interconnection topology. The global scheduler is used to dynamically adjust the distribution mapping of the next MoE layer based on the load statistics of each computing chip. Each computing core includes an input cache module, an expert weight storage module, a computing unit array, a routing index table, a D2D data transmission module, and a load monitoring and control module; The input cache module is used to store the token embedding received by this core. The expert weight storage module is used to store a subset of routing expert weights assigned to this core and local copies of shared experts; The computing unit array is used to perform GEMM operations on target experts who hit the storage range of this core particle based on the token embedding and the expert weights, to obtain a partial sum; The routing index table is used to store the mapping relationship between tokens, target experts, and core particles; The D2D data transmission module is used to transmit token embeddings along the forward direction of the ring interconnect topology and transmit a portion of the embeddings along the reverse direction of the ring interconnect topology. The load monitoring and control module is used to collect load statistics of this core and report the load statistics to the global scheduler.

10. The method of any one of claims 1-8, or the system of claim 9, in the application of inference acceleration for hybrid expert models.

Images